Method, device and electronic device for determining feature points of watermark image

Abstract

The present invention discloses a method, device and electronic device for determining feature points of a watermark image, and relates to the field of computer technology, in order to solve the problem of low detection accuracy in existing watermark detection schemes. The method includes determining matching feature points in at least two image frames in a video to be processed; obtaining a first preliminary feature point set corresponding to the watermark image with unchanged position from the matching feature points; determining an expandable point set to be confirmed and a removable point set to be confirmed based on the feature points on the convex hull of the first preliminary feature point set; determining expandable points and removable points based on the feature point coordinate errors corresponding to the watermark image part and the reprojection errors corresponding to the non-watermark image part; and obtaining a feature point set of the watermark image by adding expandable points to the first preliminary feature point set and removing removable points. The embodiment of the present invention can realize accurate detection of watermark images.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and electronic equipment for determining feature points of a watermark image. Background Art

[0002] Temporal image matching is an important basic step in the simultaneous localization and mapping (SLAM) system. For images containing watermarks or other interference items, the existing technology generally uses a mask layer to filter them out to avoid participating in the actual image matching. The mask can be generated by manually marking the watermark position or by running a target detection and segmentation neural network for detection. However, by manually marking the watermark position, it cannot be guaranteed that the labeling standard and accuracy of each image are consistent, which will affect the accuracy of the matching results to a certain extent. The method of watermark detection through the target detection network may result in missed detection and false detection, resulting in low accuracy of watermark detection.

[0003] In summary, the watermark detection scheme in the existing SLAM system has the problem of low detection accuracy. Summary of the invention

[0004] The embodiments of the present invention provide a method, a device and an electronic device for determining feature points of a watermark image, so as to solve the problem of low detection accuracy of a watermark detection solution in an existing SLAM system.

[0005] In a first aspect, an embodiment of the present invention provides a method for determining feature points of a watermark image, comprising:

[0006] Determine matching feature points in at least two image frames in the video to be processed;

[0007] From the matched feature points, a first set of initially selected feature points corresponding to the watermark image in the video to be processed is obtained; the first set of initially selected feature points is feature points whose positions remain unchanged;

[0008] Based on the feature points on the convex hull of the first preliminarily selected feature point set, determining an extensible point set to be confirmed and a removable point set to be confirmed;

[0009] Determine an extensible point from the extensible point set to be confirmed, and determine a removable point from the removable point set to be confirmed based on a feature point coordinate error corresponding to the watermark image portion and a reprojection error corresponding to the non-watermark image portion;

[0010] The first preliminary selected feature point set is corrected by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

[0011] Optionally, obtaining a first set of preliminarily selected feature points corresponding to the watermark image in the video to be processed from the matched feature points includes:

[0012] Based on the coordinate errors of the feature points matched in the i-th image frame and the i+1-th image frame in the video to be processed, determining the i+1-th fixed point set corresponding to the i+1-th image frame from the matched feature points; wherein i≥1;

[0013] According to multiple fixed point sets corresponding to multiple consecutive image frames after the i+1th image frame, unqualified feature points in the i+1th fixed point set are removed through iterative calculation to obtain a first preliminary feature point set corresponding to the watermark image; wherein the unqualified feature points are feature points that do not exist in the multiple fixed point sets corresponding to the multiple consecutive image frames.

[0014] Optionally, removing unqualified feature points in the i+1th fixed point set by iterative calculation according to the multiple fixed point sets corresponding to the multiple continuous image frames after the i+1th image frame to obtain the first preliminary feature point set corresponding to the watermark image includes:

[0015] By clustering calculation, the i+1th fixed point set is divided into N sub-point sets;

[0016] According to a plurality of fixed point sets corresponding to a plurality of continuous image frames after the i+1th image frame, a target sub-point set in the N sub-point sets is deleted through iterative calculation; wherein the number of unqualified feature points in the target sub-point set is greater than a preset value;

[0017] When the number of the deleted target sub-point sets is N-1, a sub-point set that has not been deleted among the N sub-point sets is used as the first preliminary feature point set; N is a positive integer, and N>1.

[0018] Optionally, deleting the target sub-point set in the N sub-point sets by iterative calculation according to the multiple fixed point sets corresponding to the multiple continuous image frames after the (i+1)th image frame includes:

[0019] Determine the i+2th fixed point set corresponding to the i+2th image frame;

[0020] Taking the feature points in the N sub-point sets that do not belong to the (i+2)th fixed point set as unqualified feature points in the N sub-point sets;

[0021] Deleting the target sub-point set from the N sub-point sets to obtain P sub-point sets, wherein 1≤P≤N;

[0022] When P>1, multiple fixed point sets corresponding to multiple continuous image frames after the i+2th image are iteratively calculated in sequence until P-1 sub-point sets in the P sub-point sets are deleted.

[0023] Optionally, based on the feature points on the convex hull of the first preliminarily selected feature point set, determining the extensible point set to be confirmed and the removable point set to be confirmed includes:

[0024] Constructing a triangular mesh according to all feature points in the j-th image frame in the video to be processed;

[0025] Based on the triangular mesh, using the first target feature points directly connected to each first observation point, an expandable point set to be confirmed is formed; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set, and the first target feature point is a feature point outside the first preliminary feature point set;

[0026] Based on the triangular mesh, a set of removable points to be confirmed is constructed using second target feature points directly connected to each second observation point; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the second target feature point is a feature point on the upper edge of the convex hull of the second preliminary feature point set.

[0027] Optionally, the determining of the extensible points from the extensible point set to be confirmed based on the feature point error corresponding to the watermark image portion and the reprojection error corresponding to the non-watermark image portion includes:

[0028] Determine an extensible point from the extensible point set to be confirmed according to the first value and the second value;

[0029] The first value is the coordinate error of the first preliminary feature point set before and after the expandable point is added; the second value is the reprojection error corresponding to the non-watermark feature point set before and after the expandable point is removed.

[0030] Optionally, determining the extensible point from the extensible point set to be confirmed according to the first value and the second value includes:

[0031] For the nth first observation point, after adding each of the extensible points to be confirmed to the first preliminary feature point set, a first error value is calculated according to the first value and the second value; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set;

[0032] When the smallest first error value is smaller than the second error value, the expandable point to be confirmed corresponding to the smallest first error value is determined as the expandable point; wherein the second error value is calculated based on the first preliminary feature point set corrected based on the n-1th first observation point; n≥2.

[0033] Optionally, the determining the removable points from the removable point set to be confirmed based on the feature point error corresponding to the watermark image portion and the reprojection error corresponding to the non-watermark image portion includes:

[0034] Determine a removable point from the removable point set to be confirmed according to the third value and the fourth value;

[0035] Among them, the third value is the coordinate error of the second preliminary feature point set before and after removing the removable points, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the fourth value is the reprojection error corresponding to the non-watermark feature point set before and after adding the removable points.

[0036] Optionally, the determining the removable point from the set of removable points to be confirmed according to the third value and the fourth value includes:

[0037] For the mth second observation point, after adding each of the removable points to be confirmed to the second preliminary feature point set, a third error value is calculated according to the third value and the fourth value; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set;

[0038] When the smallest third error value is smaller than the fourth error value, the to-be-confirmed removable point corresponding to the smallest third error value is determined as the removable point; wherein the fourth error value is calculated based on the second preliminary feature point set corrected based on the m-1th second observation point; m≥2.

[0039] Optionally, the first value is calculated according to the following formula:

[0040]

[0041] Among them, T Homois the coordinate error of the first preliminary feature point set before and after adding the extensible point; p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the first preliminary feature point set before updating each of the extensible points to be confirmed, and P2 is the coordinate of the first preliminary feature point set after updating each of the extensible points to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update.

[0042] Optionally, the third value is calculated according to the following formula:

[0043]

[0044] Among them, T Homo To remove the coordinate error of the second preliminary feature point set before and after the removable point, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, and P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update.

[0045] Optionally, after determining the extensible points from the extensible point set to be confirmed; and determining the removable points from the removable point set to be confirmed, the method further comprises:

[0046] After removing one of the removable points from the non-watermark feature point set each time to obtain a reduced non-watermark feature point set, updating the camera's position and the coordinates of the three-dimensional map points according to the reduced non-watermark feature point set; and / or,

[0047] After adding one of the extensible points to the non-watermark feature point set each time to obtain the expanded non-watermark feature point set, the camera's position and the three-dimensional map point coordinates are updated according to the expanded non-watermark feature point set.

[0048] In a second aspect, an embodiment of the present invention further provides a device for determining feature points of a watermark image, comprising:

[0049] A first processing module, used to determine matching feature points in at least two image frames in a video to be processed;

[0050] A second processing module is used to obtain a first set of preliminarily selected feature points corresponding to the watermark image in the video to be processed from the matched feature points; the first set of preliminarily selected feature points is feature points whose positions remain unchanged;

[0051] A third processing module, configured to determine an extensible point set to be confirmed and a removable point set to be confirmed based on the feature points on the convex hull of the first initially selected feature point set;

[0052] a fourth processing module, configured to determine an expandable point from the expandable point set to be confirmed, and determine a removable point from the removable point set to be confirmed, based on a feature point error corresponding to the watermark image portion and a reprojection error corresponding to the non-watermark image portion;

[0053] The fifth processing module is used to modify the first preliminary selected feature point set by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

[0054] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the method for determining feature points of a watermark image as described above.

[0055] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the method for determining feature points of a watermark image as described above are implemented.

[0056] In an embodiment of the present invention, matching feature points are determined in at least two image frames in a video to be processed; a first preliminary feature point set corresponding to a watermark image in the video to be processed is obtained from the matching feature points; the first preliminary feature point set is feature points with unchanged positions; based on the feature points on the convex hull of the first preliminary feature point set, an expandable point set to be confirmed and a removable point set to be confirmed are determined; based on the feature point coordinate errors corresponding to the watermark image part and the reprojection errors corresponding to the non-watermark image part, expandable points are determined from the expandable point set to be confirmed, and removable points are determined from the removable point set to be confirmed; the first preliminary feature point set is corrected by adding the expandable points to the first preliminary feature point set and removing the removable points to obtain the feature point set of the watermark image. Therefore, by using the solution of the embodiment of the present invention, it is possible to determine the first preliminary feature point set whose position remains unchanged based on the matched feature points in the image frame, and further, by calculating the feature point coordinate error corresponding to the watermark image part and the reprojection error corresponding to the non-watermark image part, the first preliminary feature point set is corrected to obtain the preliminary feature point set of the watermark image, thereby realizing accurate detection of the feature points of the watermark image. In this way, when performing real-time positioning and map construction, the influence of the watermark can be accurately eliminated by removing the feature points of the watermark image. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0058] Figure 1 is a flow chart of a method for determining characteristic points of a watermark image provided by an embodiment of the present invention;

[0059] Figure 2 is a schematic diagram of fixed point clustering provided by an embodiment of the present invention;

[0060] Figure 3 is one of the schematic diagrams of a triangular network provided by an embodiment of the present invention;

[0061] Figure 4 This is the second schematic diagram of a triangular network provided by an embodiment of the present invention;

[0062] Figure 5 is a structural diagram of a device for determining feature points of a watermark image provided by an embodiment of the present invention;

[0063] Figure 6 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] See also Figure 1 , Figure 1 is a flow chart of a method for determining feature points of a watermark image provided by an embodiment of the present invention. Figure 1 As shown, the following steps are included:

[0066] Step 101: determining matching feature points in at least two image frames in a video to be processed.

[0067] It should be noted that in image processing, feature points refer to points where the grayscale value of an image changes dramatically or points with large curvature on the edge of an image (i.e., the intersection of two edges). Image feature points can reflect the essential features of an image and identify the target object in the image. Image matching can be accomplished by matching feature points.

[0068] Feature point matching is an important method in the field of image processing to find information associations between different images. Since the imaging field of view changes due to camera movement, the same object will appear in different positions in the image. Feature point matching can quickly locate the position of the object in the new image.

[0069] In this step, the matched feature points are the same regions or pixels in at least two image frames.

[0070] Step 102: obtaining a first set of initially selected feature points corresponding to the watermark image in the video to be processed from the matched feature points; the first set of initially selected feature points is feature points whose positions remain unchanged.

[0071] It should be noted that the image frames in the video to be processed are time-series image frames recorded as the camera moves, and the position of the watermark image in each time-series image frame is fixed, while the position of the image other than the watermark image in the previous and next frames changes with the change of the camera's motion posture; wherein the position includes position and posture, and the posture is the direction in which the camera is facing. Therefore, the feature points corresponding to the watermark image must belong to the matching feature points. Based on this, the first preliminary feature point set corresponding to the feature points with unchanged positions can be determined from the matching feature points. The first preliminary feature point set is a rough estimate of the watermark image.

[0072] Step 103, determining an extensible point set to be confirmed and a removable point set to be confirmed based on the feature points on the convex hull of the first initially selected feature point set;

[0073] In this step, the first preliminary selected feature point set is a set of planar feature points in the image video frame, and the convex hull of the first preliminary selected feature point set is the smallest convex polygon containing all feature points of the first preliminary selected feature point set.

[0074] For example, Figure 3 , the dotted line connected part is the first preliminary feature point set (non-black points), and the solid line connected part is the point not in the first preliminary feature point set (black points). Among them, the 6 feature points in the outermost circle of the dotted line connected part are the feature points on the convex hull of the first preliminary feature point set. For ease of understanding and description, Figure 3 In the first preliminary feature point set, there is only one feature point in the convex hull.

[0075] Step 104, based on the coordinate error of the feature point corresponding to the watermark image part and the reprojection error corresponding to the non-watermark image part, determine the expandable points from the expandable point set to be confirmed, and determine the removable points from the removable point set to be confirmed.

[0076] In this step, the extensible points to be confirmed are feature points outside the first preliminary feature point set, and the removable points to be confirmed are feature points in the first preliminary feature point set.

[0077] Step 105: Correct the first preliminary selected feature point set by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

[0078] In this step, the correction of the first preliminary feature point set includes: adding the missing feature points (expandable points) to the first preliminary feature point set, and removing the feature points that do not belong to the watermark image (removable points) from the first preliminary feature point set. It can be understood that, for each extensible point determined, the feature point set corresponding to the watermark image part has one more feature point, while the feature point belonging to the non-watermark image part decreases by one; for each removable point determined, the feature point set corresponding to the watermark image part decreases by one feature point, while the feature point belonging to the non-watermark image part increases by one.

[0079] In the above embodiment, based on the matched feature points in the image frame, a first set of preliminarily selected feature points with unchanged positions can be determined. Furthermore, by calculating the coordinate error of the feature points corresponding to the watermark image part and the reprojection error corresponding to the non-watermark image part, the first set of preliminarily selected feature points is corrected to obtain the preliminarily selected feature point set of the watermark image, thereby realizing accurate detection of the feature points of the watermark image. In this way, when performing real-time positioning and map construction, the influence of the watermark can be accurately eliminated by removing the feature points of the watermark image.

[0080] In one embodiment, the above step 102 includes:

[0081] Based on the coordinate errors of the feature points matched in the i-th image frame and the i+1-th image frame in the video to be processed, determining the i+1-th fixed point set corresponding to the i+1-th image frame from the matched feature points; wherein i≥1;

[0082] According to multiple fixed point sets corresponding to multiple consecutive image frames after the i+1th image frame, unqualified feature points in the i+1th fixed point set are removed through iterative calculation to obtain a first preliminary feature point set corresponding to the watermark image; wherein the unqualified feature points are feature points that do not exist in the multiple fixed point sets corresponding to the multiple consecutive image frames.

[0083] In this embodiment, since the positions of the feature points corresponding to the watermark image in the two image frames are fixed, the coordinates of the feature points corresponding to the watermark image change little and are almost unchanged in the two image frames, while the coordinates of the feature points corresponding to the non-watermark image change greatly in the two frames, and this change relationship depends on the change of the camera posture. Based on this principle, this embodiment takes out the coordinates of the successfully matched points based on the matching relationship between the feature points in the i-th image frame and the i+1-th image frame, and records them as P1(X1, Y1) and P2(X2, Y2), and determines whether the horizontal and vertical coordinate errors of P1 and P2 are within a very small range; for example, X1-X2≤e and Y1-Y2≤e; where e=10 -6 ; then it is determined that the matched feature point belongs to the i+1th fixed point set corresponding to the i+1th image frame.

[0084] In one embodiment, the method of removing unqualified feature points in the i+1th fixed point set by iterative calculation based on the multiple fixed point sets corresponding to the multiple continuous image frames after the i+1th image frame to obtain the first preliminary feature point set corresponding to the watermark image includes:

[0085] By clustering calculation, the i+1th fixed point set is divided into N sub-point sets;

[0086] According to a plurality of fixed point sets corresponding to a plurality of continuous image frames after the i+1th image frame, a target sub-point set in the N sub-point sets is deleted through iterative calculation; wherein the number of unqualified feature points in the target sub-point set is greater than a preset value;

[0087] When the number of the deleted target sub-point sets is N-1, a sub-point set that has not been deleted among the N sub-point sets is used as the first preliminary feature point set; N is a positive integer, and N>1.

[0088] Exemplarily, the step of dividing the i+1th fixed point set into N sub-point sets by clustering calculation includes:

[0089] Step 1: Select N center points from the i+1th fixed point set.

[0090] Among them, N can be customized, and the value of N is the number of the final sub-point sets, recorded as C1, C2...Cn. N point sets are formed, and each point set has a center point. Figure 2 As shown, it shows the feature points in the i+1th fixed point set corresponding to the i+1th image frame. Figure 2 N=5, where the five-pointed star is the center point of each sub-point set. Figure 2 The origins of the same filling in belong to the same sub-point set.

[0091] Step 2: Calculate the distance from other non-center points to each center point to get the center point closest to itself, and add the feature point closest to itself to the point set where the nearest center point is located.

[0092] Step 3: In each sub-point set, calculate the sum of the distances from each point to the remaining points in the sub-point set in turn, and take the point with the smallest sum of distances as the new center point of the sub-point set.

[0093] Step 4: Repeat steps 3 and 4 until the center points of each sub-point set no longer change, and the final N sub-point sets can be obtained.

[0094] In a specific embodiment, deleting the target sub-point set in the N sub-point sets by iterative calculation according to the multiple fixed point sets corresponding to the multiple continuous image frames after the i+1th image frame includes:

[0095] Determine the i+2th fixed point set corresponding to the i+2th image frame;

[0096] Taking the feature points in the N sub-point sets that do not belong to the (i+2)th fixed point set as unqualified feature points in the N sub-point sets;

[0097] Deleting the target sub-point set from the N sub-point sets to obtain P sub-point sets, wherein 1≤P≤N;

[0098] When P>1, multiple fixed point sets corresponding to multiple continuous image frames after the i+2th image are iteratively calculated in sequence until P-1 sub-point sets in the P sub-point sets are deleted.

[0099] Exemplarily, deleting a target sub-point set from the N sub-point sets includes:

[0100] Perform feature point extraction, image matching, fixed point set confirmation and other steps on the next continuous image frames to obtain multiple fixed point sets corresponding to multiple continuous image frames; based on multiple fixed point sets, remove unqualified feature points in the i+1th fixed point set, and when more than a preset number (such as one-third of the points) in a sub-point set are removed, delete the sub-point set (target sub-point set). Based on multiple fixed point sets, iterate and calculate in sequence until N-1 sub-point sets are deleted, and the one sub-point set that has not been deleted is used as the first preliminary feature point set.

[0101] In one embodiment, the above step 103 includes:

[0102] Constructing a triangular mesh according to all feature points in the j-th image frame in the video to be processed;

[0103] Based on the triangular mesh, using the first target feature points directly connected to each first observation point, an expandable point set to be confirmed is formed; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set, and the first target feature point is a feature point outside the first preliminary feature point set;

[0104] Based on the triangular mesh, a set of removable points to be confirmed is constructed using second target feature points directly connected to each second observation point; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the second target feature point is a feature point on the convex hull of the second preliminary feature point set.

[0105] For example, Figure 3 , which shows a schematic diagram of a triangular mesh. The dashed line connected part is the first preliminary feature point set (non-black points), and the solid line connected part is the point in the non-first preliminary feature point set (black points). The 6 feature points in the outermost circle of the dashed line connected part are the feature points on the convex hull of the first preliminary feature point set. For ease of understanding and description, the Figure 3 In the example, there is only one feature point that is concentrated in the convex hull of the first selected feature points. Figure 3In the triangular mesh shown, when feature point a is the first observation point, feature point b, feature point c and feature point d are all first target feature points, and the expandable point set to be confirmed includes: feature point b, feature point c and feature point d.

[0106] For example, based on Figure 4 In the triangular mesh shown, feature point b is updated as an extension point to the first preliminary feature set to obtain a second preliminary feature point set; based on the second preliminary feature point set, when feature point b is the second observation point, feature point a and feature point g are both second target feature points, and the removable point set to be confirmed includes: feature point a and feature point g.

[0107] In a specific embodiment, the step of determining the scalable points from the scalable point set to be confirmed based on the feature point errors corresponding to the watermark image portion and the reprojection errors corresponding to the non-watermark image portion includes:

[0108] Determine an extensible point from the extensible point set to be confirmed according to the first value and the second value;

[0109] The first value is the coordinate error of the first preliminary feature point set before and after the expandable point is added; the second value is the reprojection error corresponding to the non-watermark feature point set before and after the expandable point is removed.

[0110] Optionally, determining the extensible point from the extensible point set to be confirmed according to the first value and the second value includes:

[0111] For the nth first observation point, after adding each of the extensible points to be confirmed to the first set of preliminary feature points, a first error value is calculated according to the first value and the second value; wherein the first observation point is a feature point on the convex hull of the first set of preliminary feature points;

[0112] When the smallest first error value is smaller than the second error value, the expandable point to be confirmed corresponding to the smallest first error value is determined as the expandable point; wherein the second error value is calculated based on the first preliminary feature point set corrected based on the n-1th first observation point; n≥2.

[0113] The first error value is the sum of the first value and the second value; the first value is calculated according to the first formula, and the second value is calculated according to the second formula.

[0114] In the above embodiment, for each first observation point, when trying to add the extensible points to be confirmed to the first preliminary feature point set, find the extensible points that can make T Homo With T ReprThe feature point with the smallest total error is used to determine whether it is an expandable point.

[0115] Optionally, the formula 1 is:

[0116]

[0117] Among them, T Homo is the coordinate error of the first preliminary feature point set before and after adding the extensible point; p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the first preliminary feature point set before updating each of the extensible points to be confirmed, and P2 is the coordinate of the first preliminary feature point set after updating each of the extensible points to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0118] It should be pointed out that in the above formula 1, when the selected feature point P is closer to the center point, the smaller the numerator of formula 1 is, the more directly connected watermark feature points are, the larger the denominator is (because the distances of such feature points need to be accumulated, the more the number, the larger the denominator is), the smaller the entire coefficient is, and the e calculated later is Homo P1-H×P2 is the error term introduced by the new extension point.

[0119] Specifically, H is the identity matrix, that is:

[0120]

[0121] In a specific embodiment, the step of determining the removable points from the set of removable points to be confirmed includes:

[0122] Determine a removable point from the removable point set to be confirmed according to the third value and the fourth value;

[0123] Among them, the third value is the coordinate error of the second preliminary feature point set before and after removing the removable points, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the fourth value is the reprojection error corresponding to the non-watermark feature point set before and after adding the removable points.

[0124] Optionally, the determining the removable point from the set of removable points to be confirmed according to the third value and the fourth value includes:

[0125] For the mth second observation point, after adding each of the removable points to be confirmed to the second preliminary feature point set, a third error value is calculated according to the third value and the fourth value; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set;

[0126] When the smallest third error value is smaller than the fourth error value, the to-be-confirmed removable point corresponding to the smallest third error value is determined as the removable point; wherein the fourth error value is calculated based on the second preliminary feature point set corrected based on the m-1th second observation point; m≥2.

[0127] Exemplarily, the third error value is the sum of a third value and a fourth value; wherein the third value is calculated according to Formula Three, and the fourth value is calculated according to Formula Four.

[0128] In the above embodiment, for each second observation point, when trying to add a point to be confirmed to be removed to the second preliminary feature point set, find the point that can make T Homo With T Repr The feature point with the smallest total error is used to determine whether the point can be removed.

[0129] Optionally, the formula three is:

[0130]

[0131] Among them, T Homo To remove the coordinate error of the second preliminary feature point set before and after the removable point, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, and P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0132] It should be pointed out that in the above formula 3, when the selected feature point P is closer to the center point, the smaller the numerator of formula 3 is, the more directly connected watermark feature points are, the larger the denominator is (because the distances of such feature points need to be accumulated, the more the number, the larger the denominator is), the smaller the entire coefficient is, and the e calculated later is Homo P1-H×P2 is the error term introduced by the new extension point.

[0133] It should be noted that, ideally, P1 = H × P2. However, in reality, since the obtained watermark feature point set may not be completely accurate, there may be a difference between P1 and H × P2. This difference is defined as T Homo =|P1-H×P2|, and this difference should theoretically be caused by the introduction of feature points that do not belong to the watermark, so when a feature point that does not belong to the watermark is correctly removed, then T Homo The value of should be significantly reduced. On the contrary, when a feature point belonging to the watermark is correctly added, T Homo The value will not increase significantly (it may increase slightly due to errors).

[0134] When trying to add a point, the watermark feature point set will increase by one point, T Homo The value of will increase, and the normal feature point belonging to the SLAM system will decrease by one. When the camera pose is determined, the entire reprojection error (denoted as T Repr ) will decrease. When the added point is the correct watermark feature point, T Homo will only increase slightly, while T Repr Will be significantly reduced.

[0135] It should be pointed out that in the specific implementation, all the expandable points should be found out based on all the first observation points of the first preliminary feature point set. After all the expandable points are determined, all the removable points should be found out based on all the second observation points of the second preliminary feature point set.

[0136] In one embodiment, after determining the extensible points from the extensible point set to be confirmed; and determining the removable points from the removable point set to be confirmed, the method further comprises:

[0137] After removing one of the removable points from the non-watermark feature point set each time to obtain a reduced non-watermark feature point set, updating the camera's position and the coordinates of the three-dimensional map points according to the reduced non-watermark feature point set; and / or,

[0138] After adding one of the extensible points to the non-watermark feature point set each time to obtain the expanded non-watermark feature point set, the camera's position and the three-dimensional map point coordinates are updated according to the expanded non-watermark feature point set.

[0139] In this embodiment, the non-watermark feature point set is a normal feature point set belonging to the SLAM system, and the feature points in the watermark feature point set will be shielded by the SLAM system and will not participate in the image matching and global optimization steps of the SLAM. This embodiment is based on the update of the feature point set of the watermark image to achieve the update of the normal point set, thereby achieving real-time update of the camera's posture and three-dimensional map point coordinates based on the normal point set.

[0140] Among them, after obtaining a set of feature points corresponding to a refined watermark image, after shielding the feature points belonging to the watermark image, re-running the conventional SLAM algorithm can accurately eliminate the influence of the watermark image and obtain a more accurate SLAM mapping or tracking result.

[0141] Next, the reprojection error T Repr Provide explanation.

[0142] Specifically, after feature point matching, epipolar constraint or PNP calculation, the conversion relationship from camera coordinates to world coordinates can be obtained, where the rotation part is recorded as R, the translation part is recorded as t, and the 3D map point in triangulation is recorded as p. H is the homography matrix between the two points, (R, t) is the conversion relationship between the two frames of camera poses, and d is the three-dimensional coordinate point (3D point) formed by the intersection in space.

[0143] Furthermore, based on the camera pose, the reprojection error T is calculated by the following formula: Repr :

[0144]

[0145] Among them, the value of i is 1 or 2, which means that the image feature points before and after the update are reprojected respectively. K is the camera intrinsic parameter, which is a known parameter. When trying to add the extensible points to be confirmed to the first preliminary feature point set, a negative sign is taken; when trying to remove the removable points to be confirmed from the second preliminary feature point set, a positive sign is taken.

[0146] Specifically, when trying to add the expandable point to be confirmed to the first preliminary feature point set, the first error value is calculated by the following objective function formula (1):

[0147]

[0148] Among them, p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the first preliminary feature point set before updating each extensible point to be confirmed, P2 is the coordinate of the first preliminary feature point set after updating each extensible point to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update. K is the camera intrinsic parameter, the rotation part of the camera position is denoted as R, the translation part of the camera position is denoted as t, x1, x2 are the feature points in the two frames respectively, d is the three-dimensional coordinate point formed by the intersection in space; e is the base of the natural logarithm.

[0149] Specifically, when trying to add the extensible point to be confirmed to the first preliminary feature point set, the third error value is calculated by the following objective function formula (2):

[0150]

[0151] Among them, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p), node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update. K is the camera internal parameter, the rotation part of the camera position is recorded as R, the translation part of the camera position is recorded as t, x1, x2 are the feature points in the two frames respectively, d is the three-dimensional coordinate point formed by the intersection in space; e is the base of the natural logarithm.

[0152] In the above embodiment, each time a feature point is selected to be added or removed, the above objective function formulas (1) and (2) are iteratively minimized, and the objective function formula converges (i.e., stabilizes near a minimum value, indicating that adding feature points or removing feature points has little effect on the objective function), and the process ends. For the successfully extended points, they are defined as new observation points, and for the successfully removed points, they are deleted from the watermark feature point set.

[0153] In the above scheme, by effectively filtering out interference items such as watermark images, the available data sets are no longer limited to self-collected data, and can widely support data sets collected online, while further improving the accuracy of feature matching and the accuracy of camera pose.

[0154] The embodiment of the present invention also provides a device for determining feature points of a watermark image. Figure 5 , Figure 5 It is a structural diagram of a feature point device for a watermark image provided by an embodiment of the present invention.

[0155] like Figure 5 As shown, the feature point determination device 500 of the watermark image includes:

[0156] A first processing module 501 is used to determine matching feature points in at least two image frames in a video to be processed;

[0157] The second processing module 502 is used to obtain a first preliminary selected feature point set corresponding to the watermark image in the video to be processed from the matched feature points; the first preliminary selected feature point set is feature points with unchanged positions;

[0158] A third processing module 503 is used to determine an extensible point set to be confirmed and a removable point set to be confirmed based on the feature points on the convex hull of the first initially selected feature point set;

[0159] A fourth processing module 504 is used to determine the extensible points from the extensible point set to be confirmed and determine the removable points from the removable point set to be confirmed based on the feature point errors corresponding to the watermark image portion and the reprojection errors corresponding to the non-watermark image portion;

[0160] The fifth processing module 505 is used to modify the first preliminary selected feature point set by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

[0161] Optionally, the second processing module 502 includes:

[0162] A first processing submodule is configured to determine, based on coordinate errors of feature points matched in an i-th image frame and an i+1-th image frame in the video to be processed, an i+1-th fixed point set corresponding to the i+1-th image frame from the matched feature points; wherein i≥1;

[0163] The second processing submodule is used to remove unqualified feature points in the i+1th fixed point set through iterative calculation according to multiple fixed point sets corresponding to multiple consecutive image frames after the i+1th image frame, so as to obtain a first preliminary feature point set corresponding to the watermark image; wherein the unqualified feature points are feature points that do not exist in the multiple fixed point sets corresponding to the multiple consecutive image frames.

[0164] Optionally, the second processing submodule includes:

[0165] A first processing unit, configured to divide the (i+1)th fixed point set into N sub-point sets by clustering calculation;

[0166] A second processing unit is used to delete a target sub-point set from the N sub-point sets through iterative calculation according to a plurality of fixed point sets corresponding to a plurality of continuous image frames after the i+1th image frame; wherein the number of unqualified feature points in the target sub-point set is greater than a preset value;

[0167] The third processing unit is used to use a sub-point set that has not been deleted from the N sub-point sets as the first preliminary feature point set when the number of the deleted target sub-point sets is N-1; N is a positive integer and N>1.

[0168] Optionally, the second processing unit is specifically configured to:

[0169] Determine the i+2th fixed point set corresponding to the i+2th image frame;

[0170] Taking the feature points in the N sub-point sets that do not belong to the (i+2)th fixed point set as unqualified feature points in the N sub-point sets;

[0171] Deleting the target sub-point set from the N sub-point sets to obtain P sub-point sets, wherein 1≤P≤N;

[0172] When P>1, multiple fixed point sets corresponding to multiple continuous image frames after the i+2th image are iteratively calculated in sequence until P-1 sub-point sets in the P sub-point sets are deleted.

[0173] Optionally, the third processing module 503 includes:

[0174] A third processing submodule is used to construct a triangular mesh according to all feature points in the j-th image frame in the video to be processed;

[0175] A fourth processing submodule is used to construct an extensible point set to be confirmed based on the triangular mesh using first target feature points directly connected to each first observation point; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set, and the first target feature point is a feature point outside the first preliminary feature point set;

[0176] The fifth processing submodule is used to form a set of removable points to be confirmed based on the triangular mesh and using second target feature points directly connected to each second observation point; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the second target feature point is a feature point on the convex hull of the second preliminary feature point set.

[0177] Optionally, the fourth processing module 504 includes:

[0178] A fourth processing unit, configured to determine an extensible point from the extensible point set to be confirmed according to the first value and the second value;

[0179] The first value is the coordinate error of the first preliminary feature point set before and after the expandable point is added; the second value is the reprojection error corresponding to the non-watermark feature point set before and after the expandable point is removed.

[0180] Optionally, the fourth processing unit is specifically configured to:

[0181] For the nth first observation point, after adding each of the extensible points to be confirmed to the first preliminary feature point set, a first error value is calculated according to the first value and the second value; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set;

[0182] When the smallest first error value is smaller than the second error value, the expandable point to be confirmed corresponding to the smallest first error value is determined as the expandable point; wherein the second error value is calculated based on the first preliminary feature point set corrected based on the n-1th first observation point; n≥2.

[0183] Optionally, the fourth processing module 504 includes:

[0184] A fifth processing unit, for determining a removable point from the set of removable points to be confirmed, using the third value and the fourth value;

[0185] Among them, the third value is the coordinate error of the second preliminary feature point set before and after removing the removable points, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the fourth value is the reprojection error corresponding to the non-watermark feature point set before and after adding the removable points.

[0186] Optionally, the fifth processing unit is specifically configured to:

[0187] For the mth second observation point, after adding each of the removable points to be confirmed to the second preliminary feature point set, a third error value is calculated according to the third value and the fourth value; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set;

[0188] When the smallest third error value is smaller than the fourth error value, the to-be-confirmed removable point corresponding to the smallest third error value is determined as the removable point; wherein the fourth error value is calculated based on the second preliminary feature point set corrected based on the m-1th second observation point; m≥2.

[0189] Optionally, the first value is calculated according to the following formula:

[0190]

[0191] Among them, T Homo is the coordinate error of the first preliminary feature point set before and after adding the extensible point; p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the first preliminary feature point set before updating each of the extensible points to be confirmed, and P2 is the coordinate of the first preliminary feature point set after updating each of the extensible points to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0192] Optionally, the third value is calculated according to the following formula:

[0193]

[0194] Among them, T Homo To remove the coordinate error of the second preliminary feature point set before and after the removable point, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, and P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0195] Optionally, the apparatus 500 further includes:

[0196] A fourth processing module is used for removing one of the removable points from the non-watermark feature point set each time to obtain a reduced non-watermark feature point set, and then updating the camera's posture and three-dimensional map point coordinates according to the reduced non-watermark feature point set; and / or,

[0197] The fifth processing module is used to add one of the extensible points to the non-watermark feature point set each time to obtain the expanded non-watermark feature point set, and then update the camera's posture and three-dimensional map point coordinates according to the expanded non-watermark feature point set.

[0198] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0199] like Figure 6 As shown, the electronic device of an embodiment of the present invention includes: a processor 600; and a memory 620 connected to the processor 600 through a bus interface, wherein the memory 620 is used to store programs and data used by the processor 600 when performing operations, and the processor 600 calls and executes the programs and data stored in the memory 620.

[0200] The processor 600 is used to read the program in the memory 620 and execute the following process:

[0201] Determine matching feature points in at least two image frames in the video to be processed;

[0202] From the matched feature points, a first set of initially selected feature points corresponding to the watermark image in the video to be processed is obtained; the first set of initially selected feature points is feature points whose positions remain unchanged;

[0203] Based on the feature points on the convex hull of the first preliminarily selected feature point set, determining an extensible point set to be confirmed and a removable point set to be confirmed;

[0204] Determine an extensible point from the extensible point set to be confirmed, and determine a removable point from the removable point set to be confirmed based on a feature point coordinate error corresponding to the watermark image portion and a reprojection error corresponding to the non-watermark image portion;

[0205] The first preliminary selected feature point set is corrected by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

[0206] Among them, Figure 6In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations. Among them, the transceiver 610 is connected to the bus interface for receiving and sending data under the control of the processor 600; the processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 may store data used by the processor 600 when performing operations.

[0207] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0208] Based on the coordinate errors of the feature points matched in the i-th image frame and the i+1-th image frame in the video to be processed, determining the i+1-th fixed point set corresponding to the i+1-th image frame from the matched feature points; wherein i≥1;

[0209] According to multiple fixed point sets corresponding to multiple consecutive image frames after the i+1th image frame, unqualified feature points in the i+1th fixed point set are removed through iterative calculation to obtain a first preliminary feature point set corresponding to the watermark image; wherein the unqualified feature points are feature points that do not exist in the multiple fixed point sets corresponding to the multiple consecutive image frames.

[0210] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0211] By clustering calculation, the i+1th fixed point set is divided into N sub-point sets;

[0212] According to a plurality of fixed point sets corresponding to a plurality of continuous image frames after the i+1th image frame, a target sub-point set in the N sub-point sets is deleted through iterative calculation; wherein the number of unqualified feature points in the target sub-point set is greater than a preset value;

[0213] When the number of the deleted target sub-point sets is N-1, a sub-point set that has not been deleted among the N sub-point sets is used as the first preliminary feature point set; N is a positive integer, and N>1.

[0214] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0215] Determine the i+2th fixed point set corresponding to the i+2th image frame;

[0216] Taking the feature points in the N sub-point sets that do not belong to the (i+2)th fixed point set as unqualified feature points in the N sub-point sets;

[0217] Deleting the target sub-point set from the N sub-point sets to obtain P sub-point sets, wherein 1≤P≤N;

[0218] When P>1, multiple fixed point sets corresponding to multiple continuous image frames after the i+2th image are iteratively calculated in sequence until P-1 sub-point sets in the P sub-point sets are deleted.

[0219] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0220] Constructing a triangular mesh according to all feature points in the j-th image frame in the video to be processed;

[0221] Based on the triangular mesh, using the first target feature points directly connected to each first observation point, an expandable point set to be confirmed is formed; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set, and the first target feature point is a feature point outside the first preliminary feature point set;

[0222] Based on the triangular mesh, a set of removable points to be confirmed is constructed using second target feature points directly connected to each second observation point; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the second target feature point is a feature point on the convex hull of the second preliminary feature point set.

[0223] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0224] Determine an extensible point from the extensible point set to be confirmed according to the first value and the second value;

[0225] The first value is the coordinate error of the first preliminary feature point set before and after the expandable point is added; the second value is the reprojection error corresponding to the non-watermark feature point set before and after the expandable point is removed.

[0226] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0227] For the nth first observation point, after adding each of the extensible points to be confirmed to the first set of preliminary feature points, a first error value is calculated according to the first value and the second value; wherein the first observation point is a feature point on the convex hull of the first set of preliminary feature points;

[0228] When the smallest first error value is smaller than the second error value, the expandable point to be confirmed corresponding to the smallest first error value is determined as the expandable point; wherein the second error value is calculated based on the first preliminary feature point set corrected based on the n-1th first observation point; n≥2.

[0229] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0230] Determine a removable point from the removable point set to be confirmed according to the third value and the fourth value;

[0231] Among them, the third value is the coordinate error of the second preliminary feature point set before and after removing the removable points, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the fourth value is the reprojection error corresponding to the non-watermark feature point set before and after adding the removable points.

[0232] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0233] For the mth second observation point, after adding each of the removable points to be confirmed to the second preliminary feature point set, a third error value is calculated according to the third value and the fourth value; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set;

[0234] When the smallest third error value is smaller than the fourth error value, the to-be-confirmed removable point corresponding to the smallest third error value is determined as the removable point; wherein the fourth error value is calculated based on the second preliminary feature point set corrected based on the m-1th second observation point; m≥2.

[0235] Optionally, the first value is calculated according to the following formula:

[0236]

[0237] Among them, T Homois the coordinate error of the first preliminary feature point set before and after adding the extensible point; p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the first preliminary feature point set before updating each of the extensible points to be confirmed, and P2 is the coordinate of the first preliminary feature point set after updating each of the extensible points to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0238] Optionally, the third value is calculated according to the following formula:

[0239]

[0240] Among them, T Homo To remove the coordinate error of the second preliminary feature point set before and after the removable point, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, and P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update; e is the base of the natural logarithm.

[0241] Optionally, the processor 600 is further configured to read the computer program and execute the following steps:

[0242] After removing one of the removable points from the non-watermark feature point set each time to obtain a reduced non-watermark feature point set, updating the camera's position and the coordinates of the three-dimensional map points according to the reduced non-watermark feature point set; and / or,

[0243] After adding one of the extensible points to the non-watermark feature point set each time to obtain the expanded non-watermark feature point set, the camera's position and the three-dimensional map point coordinates are updated according to the expanded non-watermark feature point set.

[0244] The device provided in the embodiment of the present invention can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated in this embodiment.

[0245] In addition, the computer-readable storage medium of the embodiment of the present invention is used to store a computer program, and the computer program can be executed by a processor to implement the steps of the method for determining feature points of a watermark image as described above.

[0246] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0247] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0248] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0249] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for determining feature points of a watermark image, characterized in that: include: Determine matching feature points in at least two image frames in the video to be processed; From the matched feature points, obtaining a first preliminary selected feature point set corresponding to the watermark image in the video to be processed; The first set of initially selected feature points are feature points whose positions remain unchanged; Based on the feature points on the convex hull of the first preliminarily selected feature point set, determining an extensible point set to be confirmed and a removable point set to be confirmed; Determine an extensible point from the extensible point set to be confirmed, and determine a removable point from the removable point set to be confirmed based on a feature point coordinate error corresponding to the watermark image portion and a reprojection error corresponding to the non-watermark image portion; The first preliminary selected feature point set is corrected by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

2. The method for determining feature points of a watermark image according to claim 1, characterized in that: The step of obtaining a first set of preliminarily selected feature points corresponding to the watermark image in the video to be processed from the matched feature points includes: Based on the coordinate errors of the feature points matched in the i-th image frame and the i+1-th image frame in the video to be processed, determining the i+1-th fixed point set corresponding to the i+1-th image frame from the matched feature points; wherein i≥1; According to multiple fixed point sets corresponding to multiple consecutive image frames after the i+1th image frame, unqualified feature points in the i+1th fixed point set are removed through iterative calculation to obtain a first preliminary feature point set corresponding to the watermark image; wherein the unqualified feature points are feature points that do not exist in the multiple fixed point sets corresponding to the multiple consecutive image frames.

3. The method for determining feature points of a watermark image according to claim 2, characterized in that: The method of removing unqualified feature points in the i+1th fixed point set by iterative calculation based on the multiple fixed point sets corresponding to the multiple continuous image frames after the i+1th image frame to obtain the first preliminary feature point set corresponding to the watermark image includes: By clustering calculation, the i+1th fixed point set is divided into N sub-point sets; According to a plurality of fixed point sets corresponding to a plurality of continuous image frames after the i+1th image frame, a target sub-point set in the N sub-point sets is deleted through iterative calculation; wherein the number of unqualified feature points in the target sub-point set is greater than a preset value; When the number of the deleted target sub-point sets is N-1, a sub-point set that has not been deleted among the N sub-point sets is used as the first preliminary feature point set; N is a positive integer, and N>1.

4. The method for determining feature points of a watermark image according to claim 3, characterized in that: The deleting the target sub-point set in the N sub-point sets by iterative calculation according to the multiple fixed point sets corresponding to the multiple continuous image frames after the i+1th image frame includes: Determine the i+2th fixed point set corresponding to the i+2th image frame; Taking the feature points in the N sub-point sets that do not belong to the (i+2)th fixed point set as unqualified feature points in the N sub-point sets; Deleting the target sub-point set from the N sub-point sets to obtain P sub-point sets, wherein 1≤P≤N; When P>1, multiple fixed point sets corresponding to multiple continuous image frames after the (i+2)th image are iteratively calculated in sequence until P-1 sub-point sets in the P sub-point sets are deleted.

5. The method for determining feature points of a watermark image according to claim 1, characterized in that: The determining of the extensible point set to be confirmed and the removable point set to be confirmed based on the feature points on the convex hull of the first initially selected feature point set includes: Constructing a triangular mesh according to all feature points in the j-th image frame in the video to be processed; Based on the triangular mesh, using the first target feature points directly connected to each first observation point, an expandable point set to be confirmed is formed; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set, and the first target feature point is a feature point outside the first preliminary feature point set; Based on the triangular mesh, a set of removable points to be confirmed is constructed using second target feature points directly connected to each second observation point; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the second target feature point is a feature point on the convex hull of the second preliminary feature point set.

6. The method for determining feature points of a watermark image according to claim 1, characterized in that: The step of determining the extensible points from the extensible point set to be confirmed based on the feature point errors corresponding to the watermark image portion and the reprojection errors corresponding to the non-watermark image portion comprises: Determine an extensible point from the extensible point set to be confirmed according to the first value and the second value; The first value is the coordinate error of the first preliminary feature point set before and after the expandable point is added; the second value is the reprojection error corresponding to the non-watermark feature point set before and after the expandable point is removed.

7. The method for determining feature points of a watermark image according to claim 6, characterized in that: The step of determining the extensible point from the extensible point set to be confirmed according to the first value and the second value includes: For the nth first observation point, after adding each of the extensible points to be confirmed to the first preliminary feature point set, a first error value is calculated according to the first value and the second value; wherein the first observation point is a feature point on the convex hull of the first preliminary feature point set; When the smallest first error value is smaller than the second error value, the expandable point to be confirmed corresponding to the smallest first error value is determined as the expandable point; wherein the second error value is calculated based on the first preliminary feature point set corrected based on the n-1th first observation point; n≥2.

8. The method for determining feature points of a watermark image according to claim 1, characterized in that: The determining of the removable points from the removable point set to be confirmed based on the feature point error corresponding to the watermark image part and the reprojection error corresponding to the non-watermark image part comprises: Determine a removable point from the removable point set to be confirmed according to the third value and the fourth value; Among them, the third value is the coordinate error of the second preliminary feature point set before and after removing the removable points, and the second preliminary feature point set includes the first preliminary feature point set and all the expandable points; the fourth value is the reprojection error corresponding to the non-watermark feature point set before and after adding the removable points.

9. The method for determining feature points of a watermark image according to claim 8, characterized in that: Determining the removable points from the set of removable points to be confirmed according to the third value and the fourth value includes: For the mth second observation point, after adding each of the removable points to be confirmed to the second preliminary feature point set, a third error value is calculated according to the third value and the fourth value; wherein the second observation point is a feature point on the convex hull of the second preliminary feature point set; When the smallest third error value is smaller than the fourth error value, the to-be-confirmed removable point corresponding to the smallest third error value is determined as the removable point; wherein the fourth error value is calculated based on the second preliminary feature point set corrected based on the m-1th second observation point; m≥2.

10. The method for determining feature points of a watermark image according to claim 6, characterized in that: The first value is calculated according to the following formula: Among them, T Homo is the coordinate error of the first preliminary feature point set before and after adding the extensible point; p is the extensible point to be confirmed; conn(p) is the feature point in the first preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the first preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set, and the second preliminary feature point set includes the first preliminary feature point set and all the extensible points; P1 is the coordinate of the first preliminary feature point set before updating each of the extensible points to be confirmed, and P2 is the coordinate of the first preliminary feature point set after updating each of the extensible points to be confirmed; H is the transformation homography matrix between the first preliminary feature point set before and after the update; e is the base of the natural logarithm.

11. The method for determining feature points of a watermark image according to claim 8, characterized in that: The third value is calculated according to the following formula: Among them, T Homo To remove the coordinate error of the second preliminary feature point set before and after the removable point, p is the removable point to be confirmed; conn(p) is the feature point in the second preliminary feature point set directly connected to p, node(p) is the hierarchical distance from p to the center point of the second preliminary feature point set; q belongs to the feature point in conn(p); node(q) is the hierarchical distance from q to the center point of the second preliminary feature point set; P1 is the coordinate of the second preliminary feature point set before updating each removable point to be confirmed, and P2 is the coordinate of the second preliminary feature point set after updating each removable point to be confirmed; H is the transformation homography matrix between the second preliminary feature point set before and after the update; e is the base of the natural logarithm.

12. The method for determining feature points of a watermark image according to claim 1, characterized in that: Determining the extensible point from the set of extensible points to be confirmed; And, after determining the removable points from the set of removable points to be confirmed, the method further includes: After removing one of the removable points from the non-watermark feature point set each time to obtain a reduced non-watermark feature point set, updating the camera's position and the coordinates of the three-dimensional map points according to the reduced non-watermark feature point set; and / or, After adding one of the extensible points to the non-watermark feature point set each time to obtain the expanded non-watermark feature point set, the camera's position and the three-dimensional map point coordinates are updated according to the expanded non-watermark feature point set.

13. A device for determining feature points of a watermark image, characterized in that: include: A first processing module, used to determine matching feature points in at least two image frames in a video to be processed; A second processing module is used to obtain a first preliminary feature point set corresponding to the watermark image in the video to be processed from the matched feature points; The first set of initially selected feature points are feature points whose positions remain unchanged; A third processing module, configured to determine an extensible point set to be confirmed and a removable point set to be confirmed based on the feature points on the convex hull of the first initially selected feature point set; a fourth processing module, configured to determine an expandable point from the expandable point set to be confirmed, and determine a removable point from the removable point set to be confirmed, based on a feature point error corresponding to the watermark image portion and a reprojection error corresponding to the non-watermark image portion; The fifth processing module is used to modify the first preliminary selected feature point set by adding the extensible points to the first preliminary selected feature point set and removing the removable points to obtain the feature point set of the watermark image.

14. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the method for determining feature points of a watermark image as described in any one of claims 1 to 12.

15. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the method for determining feature points of a watermark image according to any one of claims 1 to 12 are implemented.

Images