A three-dimensional coordinate measurement method and system based on spot tracking

Abstract

The present invention relates to a three-dimensional coordinate measurement method and system based on spot tracking, comprising selecting an initial image from a pre-acquired image pair with epipolar geometric constraints, extracting the initial spot of a ground feature, and calculating an initial offset value between the initial spot and the centroid of the corresponding tracking spot in the image pair; performing a secondary calibration on the coordinates of the corner points selected in the initial image and their corresponding matching points; and calculating the three-dimensional coordinates of all corner points based on the calibrated point pairs and left and right camera parameters to obtain the three-dimensional coordinates of the ground feature. The present invention extracts the initial spot of the initial image, calculates the initial offset value between the centroid of the corresponding tracking spot in the image pair, performs a secondary calibration on the coordinates of the pre-selected corner points and their matching points in the initial image, and finally calculates the three-dimensional coordinates of all corner points to obtain the three-dimensional coordinates of the ground feature. This method does not require the use of equipment such as 3D glasses, greatly improving the convenience of three-dimensional coordinate measurement of ground features and ensuring measurement accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of photogrammetry, and in particular to a three-dimensional coordinate measurement method and system based on spot tracking. Background Art

[0002] Photogrammetry of land features often requires measuring their three-dimensional coordinates. To obtain elevation values ​​in these three-dimensional coordinates, surveyors wear 3D glasses and adjust the depth of field within the stereoscopic view formed by the image pair. This measurement process requires specialized equipment and requires a certain level of professional competence. Even so, due to differences in perspective, professional skills, image perception, experience, and judgment, measurement results often exhibit certain deviations, significantly impacting the three-dimensional coordinate measurement of land features during photogrammetry. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and provide a three-dimensional coordinate measurement method and system based on pattern tracking.

[0004] The present invention solves the above-mentioned technical problem with the following technical solution: A three-dimensional coordinate measurement method based on pattern tracking comprises the following steps:

[0005] An initial image is selected from a pre-acquired image pair with epipolar geometric constraints, and an initial patch of a ground feature is extracted based on the initial image, and an initial offset value between the initial patch and the centroid of the corresponding tracking patch in the image pair is calculated;

[0006] Based on the initial offset value, performing secondary calibration on the pre-selected corner points in the initial image and the coordinates of the corresponding matching points;

[0007] The three-dimensional coordinates of all corner points are calculated based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground feature.

[0008] The beneficial effects of the present invention are as follows: the three-dimensional coordinate measurement method based on spot tracking of the present invention extracts the initial spot of the selected initial image in the image pair, and calculates the initial offset value between the initial spot and the centroid of the corresponding tracking spot in its image pair, and then performs a secondary calibration on the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset value, and finally calculates the three-dimensional coordinates of all corner points based on the calibrated points and the left and right camera parameters to obtain the three-dimensional coordinates of the land feature, without the need for 3D glasses and other equipment, greatly improving the convenience of three-dimensional coordinate measurement of land feature elements, overcoming the deviation during manual measurement, and ensuring measurement accuracy.

[0009] On the basis of the above technical solution, the present invention can also be improved as follows:

[0010] Further, the calculation of the initial offset value between the initial spot and the centroid of the corresponding spot in its image pair specifically includes the following steps:

[0011] The initial spot is used as the target spot to be tracked for the corresponding ground feature, and the target spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking spot in the image pair of the initial spot;

[0012] The horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot is calculated and used as the initial offset value.

[0013] The beneficial effect of the above further scheme is: by taking the initial map spot as the target map spot that needs to be tracked for the corresponding ground feature, and then using the nuclear line image pair tracking model to perform tracking processing in the image pair of the initial image, the tracking map spot can be accurately obtained, which can facilitate the subsequent determination of the initial offset value between the initial map spot and the tracking map spot.

[0014] Further: the tracking process of the target spot in the image pair of the initial image using the epipolar line image pair tracking model specifically includes the following steps:

[0015] The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with epipolar geometric constraints to obtain the image features of the image pairs.

[0016] Use the prompt encoding layer to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature;

[0017] The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

[0018] The beneficial effect of the above further scheme is: by performing image feature extraction processing on the image pair respectively, the image features of the image pair are obtained, and the attention interaction fusion processing is performed on the prompt features extracted from the initial image, and finally the tracking spot corresponding to the initial spot is decoded to obtain the initial offset value between the initial spot and the center of mass of the corresponding tracking spot in its image pair.

[0019] Further: the secondary calibration of the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset values ​​specifically includes the following steps:

[0020] A first point set P1 is constructed based on the pre-selected corner points in the initial image, and all points in the first point set P1 are offset in the X-axis direction using the initial offset value. A second point set P2 is constructed based on the points obtained after the offset processing. The calculation formula for the offset processing is:

[0021]

[0022] offset(x)=x1-x2

[0023] Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value.

[0024] Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points;

[0025] The three-dimensional coordinates of all corner points are calculated using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

[0026] The beneficial effect of the above further scheme is: the first point set P1 constructed by the corner points in the initial image is offset by the initial offset value, and then the points in the obtained second point set P2 are matched with the corresponding points in the first point set P1. Finally, based on the obtained third point set P3 and the first point set P1 combined with the parameters of the left and right cameras, the three-dimensional coordinates of all corner points are calculated, the three-dimensional coordinates of the ground features are accurately determined, and the three-dimensional coordinate measurement of the ground features is realized.

[0027] Further, the step of searching for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction specifically includes the following steps:

[0028] Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window;

[0029] Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point with the most similar eigenvalue feat1 is the midpoint of the first point set P1 matching points; the feature value of the point in the second point set P2 is the pixel value of the image window centered on the point.

[0030] The beneficial effect of the above further scheme is: by selecting an image window with a point in the first point set P1 as the center in the initial image, and then determining the matching point of the point in the first point set P1 based on the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1, thereby facilitating the subsequent calculation of the three-dimensional coordinates of the corner point.

[0031] Further: the determination of the characteristic value feat2 in the second point set P2 and the point The specific steps of finding the most similar point of eigenvalue feat1 include:

[0032] Calculate the gssim value between the eigenvalue feat1 and the eigenvalue feat2. The calculation formula is:

[0033]

[0034] in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image;

[0035] Select the point corresponding to the eigenvalue feat2 with the largest gssim value of the eigenvalue feat1 as the point in the first point set P1 The most similar point.

[0036] The beneficial effect of the above further scheme is: by calculating the gssim value between the eigenvalue feat1 and the eigenvalue feat2, the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1 is determined, thereby facilitating the determination of the best matching point.

[0037] The present invention also provides a three-dimensional coordinate measurement system based on spot tracking, comprising an extraction calculation module, a calibration module and a three-dimensional calculation module;

[0038] The extraction and calculation module is used to select an initial image from a pre-acquired epipolar geometrically constrained image pair, extract an initial spot of a ground feature based on the initial image, and calculate an initial offset value between the initial spot and the centroid of the corresponding tracking spot in the image pair;

[0039] The calibration module is configured to perform secondary calibration on the pre-selected corner points in the initial image and the coordinates of the corresponding matching points based on the initial offset values;

[0040] The three-dimensional calculation module is used to calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground features.

[0041] On the basis of the above technical solution, the present invention can also be improved as follows:

[0042] Further: the specific implementation of the extraction calculation module calculating the initial offset value between the initial image spot and the centroid of the corresponding image spot in the image pair is:

[0043] The initial spot is used as the target spot to be tracked for the corresponding ground feature, and the target spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking spot in the image pair of the initial spot;

[0044] The horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot is calculated and used as the initial offset value.

[0045] The beneficial effect of the above further scheme is: by taking the initial map spot as the target map spot that needs to be tracked for the corresponding ground feature, and then using the nuclear line image pair tracking model to perform tracking processing in the image pair of the initial image, the tracking map spot can be accurately obtained, which can facilitate the subsequent determination of the initial offset value between the initial map spot and the tracking map spot.

[0046] Further: the specific implementation of the extraction calculation module tracking the target spot in the image pair of the initial image using the epipolar line image pair tracking model is as follows:

[0047] The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with epipolar geometric constraints to obtain the image features of the image pairs.

[0048] Use the prompt encoding layer to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature;

[0049] The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

[0050] The beneficial effect of the above further scheme is: by performing image feature extraction processing on the image pair respectively, the image features of the image pair are obtained, and the attention interaction fusion processing is performed on the prompt features extracted from the initial image, and finally the tracking spot corresponding to the initial spot is decoded to obtain the initial offset value between the initial spot and the center of mass of the corresponding tracking spot in its image pair.

[0051] Further: the calibration module performs secondary calibration on the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset values ​​as follows:

[0052] A first point set P1 is constructed based on the pre-selected corner points in the initial image, and all points in the first point set P1 are offset in the X-axis direction using the initial offset value. A second point set P2 is constructed based on the points obtained after the offset processing. The calculation formula for the offset processing is:

[0053]

[0054] offset(x)=x1-x2

[0055] Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value.

[0056] Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points;

[0057] The three-dimensional coordinates of all corner points are calculated using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

[0058] The beneficial effect of the above further scheme is: the first point set P1 constructed by the corner points in the initial image is offset by the initial offset value, and then the points in the obtained second point set P2 are matched with the corresponding points in the first point set P1. Finally, based on the obtained third point set P3 and the first point set P1 combined with the parameters of the left and right cameras, the three-dimensional coordinates of all corner points are calculated, the three-dimensional coordinates of the ground features are accurately determined, and the three-dimensional coordinate measurement of the ground features is realized.

[0059] Further: the calibration module searches for matching points corresponding to points in the first point set P1 within a preset range in the X-axis direction in a specific implementation as follows:

[0060] Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window;

[0061] Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point corresponding to the eigenvalue feat1 that is most similar to the eigenvalue is the midpoint of the first point set P1 matching points; the feature value of the point in the second point set P2 is the pixel value of the image window centered on the point.

[0062] The beneficial effect of the above further scheme is: by selecting an image window with a point in the first point set P1 as the center in the initial image, and then determining the matching point of the point in the first point set P1 based on the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1, thereby facilitating the subsequent calculation of the three-dimensional coordinates of the corner point.

[0063] Further: the calibration module determines the eigenvalue feat2 in the second point set P2 and the point The specific implementation of the point with the most similar eigenvalue feat1 is:

[0064] Calculate the gssim value between the eigenvalue feat1 and the eigenvalue feat2. The calculation formula is:

[0065]

[0066] in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image;

[0067] Select the eigenvalue feat2 with the largest gssim value with the eigenvalue feat1 as the point in the first point set P1 The most similar point.

[0068] The beneficial effect of the above further scheme is: by calculating the gssim value between the eigenvalue feat1 and the eigenvalue feat2, the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1 is determined, thereby facilitating the determination of the best matching point.

[0069] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional coordinate measurement method based on pattern tracking.

[0070] The present invention also provides a three-dimensional coordinate measurement device based on pattern tracking, comprising a communication interface, a memory, a communication bus, and a processor, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0071] The memory is used to store computer programs;

[0072] The processor is used to implement the steps of the three-dimensional coordinate measurement method based on spot tracking when executing the program stored in the memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 1 is a flow chart of a method for three-dimensional coordinate measurement based on spot tracking according to an embodiment of the present invention;

[0074] Figure 2a is a schematic diagram of an example of a left image in an image pair according to an embodiment of the present invention;

[0075] Figure 2b is a schematic diagram of an example of a right image in an image pair according to an embodiment of the present invention;

[0076] Figure 3 Schematic diagram of an example of an initial patch of a land feature according to an embodiment of the present invention;

[0077] Figure 4 Schematic diagram of an example of tracking pattern of land features according to an embodiment of the present invention;

[0078] Figure 5 A schematic diagram of matching results of pre-selected corner points and their corresponding matching points in an initial image of a ground feature according to an embodiment of the present invention;

[0079] Figure 6 2 is a flow chart of a three-dimensional coordinate measurement method based on spot tracking according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0081] like Figure 1 As shown, a three-dimensional coordinate measurement method based on spot tracking includes the following steps:

[0082] S1: Selecting an initial image from a pre-acquired image pair with epipolar geometric constraints, extracting an initial spot of a ground feature based on the initial image, and calculating an initial offset value between the initial spot and the centroid of the corresponding tracking spot in the image pair;

[0083] S2: Based on the initial offset value, performing secondary calibration on the pre-selected corner points in the initial image and the coordinates of the corresponding matching points;

[0084] S3: Calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground feature.

[0085] The three-dimensional coordinate measurement method based on spot tracking of the present invention extracts the initial spot of the selected initial image in the image pair, calculates the initial offset value between the initial spot and the centroid of the corresponding tracking spot in its image pair, and then performs a secondary calibration on the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset value. Finally, the three-dimensional coordinates of all corner points are calculated based on the calibrated points and the left and right camera parameters to obtain the three-dimensional coordinates of the land feature. Without the aid of 3D glasses and other equipment, the convenience of measuring the three-dimensional coordinates of the land feature is greatly improved, the deviation during manual measurement is overcome, and the measurement accuracy is guaranteed.

[0086] In one or more embodiments of the present invention, calculating the initial offset value between the initial spot and the centroid of the corresponding spot in the image pair specifically includes the following steps:

[0087] S11: using the initial image spot as a target image spot to be tracked for the corresponding ground feature, and tracking the target image spot in the image pair of the initial image using an epipolar image pair tracking model to obtain the corresponding tracking image spot in the image pair of the initial image spot;

[0088] S12: Calculate the horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot, and use it as the initial offset value.

[0089] By taking the initial spot as the target spot that needs to be tracked for the corresponding ground feature, and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracking spot can be accurately obtained, which can facilitate the subsequent determination of the initial offset value between the initial spot and the tracking spot.

[0090] In one or more embodiments of the present invention, in an image pair constrained by epipolar geometry, the left image is selected as the initial image, and the initial patch of the feature is extracted from the left image. The image pair (i.e., the left and right images) is treated as a two-frame video, with the left image being the first frame. The initial patch of the feature is the target patch to be tracked. This patch is then processed using the epipolar image pair tracking model to obtain the patch of the feature in the right image.

[0091] In the embodiment of the present invention, Figure 2a and Figure 2b Taking the image pair shown in the figure as an example, a detailed description of three-dimensional coordinate measurement is given.

[0092] Taking the left image as the initial image, manually measure a certain feature to obtain the initial map of the feature, such as Figure 3 Shown in the red area.

[0093] Specifically, in one or more embodiments of the present invention, the tracking process of the target spot in the image pair of the initial image using the epipolar image pair tracking model specifically includes the following steps:

[0094] S111: performing image feature extraction processing on the image pairs with the epipolar line geometric constraints using an image encoding layer composed of a convolutional neural network to obtain image features of the image pairs;

[0095] S112: Encode the prompt of the point or mask in the initial image of the image pair using the prompt coding layer to obtain a prompt feature;

[0096] S113: Input the image features and prompt features into the Transformer layer for attention interaction fusion processing, and transmit them to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

[0097] The image features of the image pair are obtained by performing image feature extraction processing on each image pair, and attention interaction fusion processing is performed on the prompt features extracted from the initial image. Finally, the tracking spot corresponding to the initial spot is obtained by decoding, so as to accurately determine the initial offset value between the initial spot and the center of mass of the corresponding tracking spot in the image pair.

[0098] In practice, the left image in the image pair (e.g. Figure 2a ) as the first frame, and the right image in the image pair (as Figure 2b shown) as the second frame, Figure 3 The ground feature pattern in the first frame is used as the initial marking pattern, which is input into the epipolar image pair tracking model, and the initial marking pattern is output as the pattern in the second frame, i.e. the right image, as shown in Figure 4 The red area is shown.

[0099] In one or more embodiments of the present invention, when calculating the horizontal coordinate offset value between the initial image spot and the tracking image spot, according to the epipolar line geometric constraint, the same-name points of the image pair are conjugate and have the same vertical coordinates, so only the horizontal coordinate offset value of the center of mass is taken as the initial offset value.

[0100] In one or more embodiments of the present invention, performing secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset values ​​specifically includes the following steps:

[0101] S21: Construct a first point set P1 based on the pre-selected corner points in the initial image, and use the initial offset value to offset all points in the first point set P1 in the X-axis direction, and construct a second point set P2 based on the points obtained after the offset processing; the calculation formula for the offset processing is:

[0102]

[0103] offset(x)=x1-x2

[0104] Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value.

[0105] In practice, the two-dimensional centroid coordinates (x1, y1) of the image spot in the left image and the two-dimensional centroid coordinates (x2, y2) of the image spot in the right image are calculated. Since the left and right images satisfy the epipolar geometric constraint, the same name in the image pair only exists on the x-axis, that is, the offset in the horizontal coordinate direction, so only the offset value of the horizontal coordinate offset(x)=x1-x2 is calculated.

[0106] Specifically, the coordinates of all corner points manually extracted in the left image are taken as a point set P1, the number of corner points is n, and for each of these n points, an offset offset(x) in the x-axis direction is added to form a new point set P2.

[0107] S22: Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points;

[0108] S23: Calculate the three-dimensional coordinates of all corner points using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

[0109] The first point set P1 constructed by the corner points in the initial image is offset by the initial offset value, and then the points in the obtained second point set P2 are matched with the corresponding points in the first point set P1. Finally, the three-dimensional coordinates of all corner points are calculated based on the obtained third point set P3 and the first point set P1 combined with the parameters of the left and right cameras, and the three-dimensional coordinates of the ground features are accurately determined to achieve three-dimensional coordinate measurement of the ground features.

[0110] In one or more embodiments of the present invention, searching for matching points corresponding to points in the first point set P1 within a preset range in the X-axis direction specifically includes the following steps:

[0111] S221: Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window;

[0112] S222: Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point with the most similar eigenvalue feat1 is the midpoint of the first point set P1 matching points; the feature value of the point in the second point set P2 is the pixel value of the image window centered on the point.

[0113] Specifically, traverse the points in P1, recorded as , in the left image, the point Take an 11x11 image window as the center, and use the pixel value of the window as the feature of the point, recorded as feat1, and use the pixel value of P2 that is related to the point as the feature of the point. Corresponding points As the initial value, we traverse all points within the range [-5, 5] on the x-axis. The feature of each point is also the pixel value of an 11x11 window centered at that point. There are 11 points in this range, so there are 11 feature values. By comparing with feat1, we find the feature value that is most similar to feat1. The point where this feature value is located is recorded as the matching point after secondary calibration and recorded in P3.

[0114] By selecting an image window with a point in the first point set P1 as the center in the initial image, and then determining the matching point of the point in the first point set P1 based on the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1, it is convenient to subsequently calculate the three-dimensional coordinates of the corner point.

[0115] In one or more embodiments of the present invention, the determination of the characteristic value feat2 in the second point set P2 and the point The specific steps of finding the most similar point of eigenvalue feat1 include:

[0116] S2221: Calculate the gssim value between the eigenvalues ​​feat1 and feat2. The calculation formula is:

[0117]

[0118] in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image;

[0119] S2222: Select the point corresponding to the eigenvalue feat2 with the largest gssim value of the eigenvalue feat1 as the point in the first point set P1 The most similar point.

[0120] In the present invention, the gssim value between the two eigenvalues ​​is calculated and the maximum gssim value is taken. The larger the gssim value, the more similar the structure. After matching the corner points of the first point set P1 and the third point set P3, n corresponding matching points are obtained. The matching results are as follows: Figure 5 shown.

[0121] The similarity between the eigenvalue feat2 of the corresponding point after the offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1 is determined by calculating the gssim value between the eigenvalue feat1 and the eigenvalue feat2, thereby facilitating the determination of the best matching point.

[0122] In the present invention, after obtaining the best matching point, the pre-selected corner point and its corresponding matching point coordinates are calibrated twice, and then the three-dimensional coordinates of the corner point of the feature are calculated based on the parameters of the left and right cameras. The specific calculation process is the existing technology and will not be repeated here.

[0123] like Figure 6 As shown, the present invention also provides a three-dimensional coordinate measurement system based on spot tracking, including an extraction calculation module, a calibration module and a three-dimensional calculation module;

[0124] The extraction and calculation module is used to select an initial image from a pre-acquired epipolar geometrically constrained image pair, extract an initial spot of a ground feature based on the initial image, and calculate an initial offset value between the initial spot and the centroid of the corresponding tracking spot in the image pair;

[0125] The calibration module is configured to perform secondary calibration on the pre-selected corner points in the initial image and the coordinates of the corresponding matching points based on the initial offset values;

[0126] The three-dimensional calculation module is used to calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground features.

[0127] In one or more embodiments of the present invention, the extraction calculation module calculates the initial offset value between the initial spot and the centroid of the corresponding spot in its image pair as follows:

[0128] The initial spot is used as the target spot to be tracked for the corresponding ground feature, and the target spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking spot in the image pair of the initial spot;

[0129] The horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot is calculated and used as the initial offset value.

[0130] By taking the initial spot as the target spot that needs to be tracked for the corresponding ground feature, and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracking spot can be accurately obtained, which can facilitate the subsequent determination of the initial offset value between the initial spot and the tracking spot.

[0131] In one or more embodiments of the present invention, the extraction calculation module uses the epipolar image pair tracking model to track the target spot in the image pair of the initial image in the following specific implementations:

[0132] The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with epipolar geometric constraints to obtain the image features of the image pairs.

[0133] Use the prompt encoding layer to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature;

[0134] The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

[0135] The image features of the image pair are obtained by performing image feature extraction processing on each image pair, and attention interaction fusion processing is performed on the prompt features extracted from the initial image. Finally, the tracking spot corresponding to the initial spot is obtained by decoding, so as to accurately determine the initial offset value between the initial spot and the center of mass of the corresponding tracking spot in the image pair.

[0136] In one or more embodiments of the present invention, the calibration module performs secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset values ​​as follows:

[0137] A first point set P1 is constructed based on the pre-selected corner points in the initial image, and all points in the first point set P1 are offset in the X-axis direction using the initial offset value. A second point set P2 is constructed based on the points obtained after the offset processing. The calculation formula for the offset processing is:

[0138]

[0139] offset(x)=x1-x2

[0140] Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value.

[0141] Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points;

[0142] The three-dimensional coordinates of all corner points are calculated using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

[0143] The first point set P1 constructed by the corner points in the initial image is offset by the initial offset value, and then the points in the obtained second point set P2 are matched with the corresponding points in the first point set P1. Finally, the three-dimensional coordinates of all corner points are calculated based on the obtained third point set P3 and the first point set P1 combined with the parameters of the left and right cameras, and the three-dimensional coordinates of the ground features are accurately determined to achieve three-dimensional coordinate measurement of the ground features.

[0144] In one or more embodiments of the present invention, the calibration module searches for matching points corresponding to points in the first point set P1 within a preset range in the X-axis direction as follows:

[0145] Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window;

[0146] Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point with the most similar eigenvalue feat1 is the midpoint of the first point set P1 matching points; the feature value of the point in the second point set P2 is the pixel value of the image window centered on the point.

[0147] By selecting an image window with a point in the first point set P1 as the center in the initial image, and then determining the matching point of the point in the first point set P1 based on the similarity between the eigenvalue feat2 of the corresponding point after offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1, it is convenient to subsequently calculate the three-dimensional coordinates of the corner point.

[0148] In one or more embodiments of the present invention, the calibration module determines the eigenvalues ​​feat2 in the second point set P2 and the point The specific implementation of the point with the most similar eigenvalue feat1 is:

[0149] Calculate the gssim value between the eigenvalue feat1 and the eigenvalue feat2. The calculation formula is:

[0150]

[0151] in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image;

[0152] Select the point corresponding to the eigenvalue feat2 with the largest gssim value of the eigenvalue feat1 as the point in the first point set P1 The most similar point.

[0153] The similarity between the eigenvalue feat2 of the corresponding point after the offset processing in the second point set P2 and the eigenvalue feat1 of the point in the first point set P1 is determined by calculating the gssim value between the eigenvalue feat1 and the eigenvalue feat2, thereby facilitating the determination of the best matching point.

[0154] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional coordinate measurement method based on pattern tracking.

[0155] The present invention also provides a three-dimensional coordinate measurement device based on pattern tracking, comprising a communication interface, a memory, a communication bus, and a processor, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0156] The memory is used to store computer programs;

[0157] The processor is used to implement the steps of the three-dimensional coordinate measurement method based on spot tracking when executing the program stored in the memory.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A three-dimensional coordinate measurement method based on spot tracking, characterized in that: The steps include: An initial image is selected from a pre-acquired image pair with epipolar geometric constraints, and an initial patch of a ground feature is extracted based on the initial image, and an initial offset value between the initial patch and the centroid of the corresponding tracking patch in the image pair is calculated; Calibrate the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset values; Calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground features; Calculating the initial offset value between the initial spot and the centroid of the corresponding spot in its image pair specifically includes the following steps: The initial spot is used as the target spot to be tracked for the corresponding ground feature, and the target spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking spot in the image pair of the initial spot; The horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot is calculated and used as the initial offset value.

2. The three-dimensional coordinate measurement method based on spot tracking according to claim 1, characterized in that: The tracking process of the target spot in the image pair of the initial image using the epipolar image pair tracking model specifically includes the following steps: The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with epipolar geometric constraints to obtain the image features of the image pairs. Use the prompt encoding layer to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature; The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

3. The three-dimensional coordinate measurement method based on spot tracking according to claim 1, characterized in that: The step of calibrating the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset values ​​specifically includes the following steps: A first point set P1 is constructed based on the pre-selected corner points in the initial image, and all points in the first point set P1 are offset in the X-axis direction using the initial offset value. A second point set P2 is constructed based on the points obtained after the offset processing. The calculation formula for the offset processing is: ； offset(x)=x1-x2; Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value. Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points; The three-dimensional coordinates of all corner points are calculated using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

4. The three-dimensional coordinate measurement method based on spot tracking according to claim 3, characterized in that: The step of searching for matching points corresponding to points in the first point set P1 within a preset range in the X-axis direction specifically includes the following steps: Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window; Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point with the most similar eigenvalue feat1 is the midpoint of the first point set P1 The matching points of the second point set P2 are characterized by the pixel value of the image window centered at the point. The determination of the eigenvalue feat2 in the second point set P2 and the point The specific steps of finding the most similar point of eigenvalue feat1 include: Calculate the gssim value between the eigenvalue feat1 and the eigenvalue feat2. The calculation formula is: ； in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image; Select the point corresponding to the eigenvalue feat2 with the largest gssim value of the eigenvalue feat1 as the point in the first point set P1 The most similar point.

5. A three-dimensional coordinate measurement system based on pattern tracking, characterized in that: It includes extraction calculation module, calibration module and three-dimensional calculation module; The extraction and calculation module is used to select an initial image from a pre-acquired epipolar geometrically constrained image pair, extract an initial spot of a ground feature based on the initial image, and calculate an initial offset value between the initial spot and the centroid of the corresponding tracking spot in the image pair; The calibration module is configured to calibrate the pre-selected corner points in the initial image and the coordinates of the corresponding matching points based on the initial offset values; The three-dimensional calculation module is used to calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground features; The specific implementation of the extraction calculation module calculating the initial offset value between the initial spot and the centroid of the corresponding spot in the image pair is: The initial spot is used as the target spot to be tracked for the corresponding ground feature, and the target spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking spot in the image pair of the initial spot; The horizontal coordinate offset value between the initial pattern spot and the tracking pattern spot is calculated and used as the initial offset value.

6. The three-dimensional coordinate measurement system based on spot tracking according to claim 5, characterized in that: The specific implementation of the extraction calculation module tracking the target spot in the image pair of the initial image using the epipolar line image pair tracking model is as follows: The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with epipolar geometric constraints to obtain the image features of the image pairs. Use the prompt encoding layer to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature; The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

7. The three-dimensional coordinate measurement system based on spot tracking according to claim 5, characterized in that: The calibration module calibrates the pre-selected corner points in the initial image and their corresponding matching point coordinates based on the initial offset values ​​as follows: A first point set P1 is constructed based on the pre-selected corner points in the initial image, and all points in the first point set P1 are offset in the X-axis direction using the initial offset value. A second point set P2 is constructed based on the points obtained after the offset processing. The calculation formula for the offset processing is: ； offset(x)=x1-x2; Where x1 is the horizontal coordinate of the 2D centroid of the initial spot in the initial image, x2 is the horizontal coordinate of the 2D centroid of the tracking spot in the image pair of the initial image, and offset(x) is the initial offset value. Using the coordinates of the points in the second point set P2 as initial coordinates, searching for matching points with corresponding points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points; The three-dimensional coordinates of all corner points are calculated using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

8. The three-dimensional coordinate measurement system based on spot tracking according to claim 7, characterized in that: The calibration module searches for matching points corresponding to points in the first point set P1 within a preset range in the X-axis direction in a specific implementation as follows: Traverse the points in the first point set P1 , in the initial image, the point Select the image window as the center; the midpoint of the first point set P1 The characteristic value feat1 is the pixel value of the image window; Take the second point set P2 and the point Corresponding points after offset processing As the initial value, traverse all points in the preset range in the X-axis direction, and determine the eigenvalue feat2 in the second point set P2 and the point The point corresponding to the eigenvalue feat1 that is most similar to the eigenvalue is the midpoint of the first point set P1 The matching points of the second point set P2 are characterized by the pixel value of the image window centered at the point. The calibration module determines the eigenvalue feat2 in the second point set P2 and the point The specific implementation of the point with the most similar eigenvalue feat1 is: Calculate the gssim value between the eigenvalue feat1 and the eigenvalue feat2. The calculation formula is: ； in, Represents the average brightness of the points corresponding to the eigenvalue feat1 in the initial image, Represents the average brightness of the points corresponding to the eigenvalue feat2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat1 and eigenvalue feat2, C1 and C2 are constants, is the gradient vector of the point corresponding to the eigenvalue feat1 in the initial image, is the gradient vector of the point corresponding to the eigenvalue feat2 in the initial image; Select the eigenvalue feat2 with the largest gssim value with the eigenvalue feat1 as the point in the first point set P1 The most similar point.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the three-dimensional coordinate measurement method based on spot tracking described in any one of claims 1 to 4 is implemented.

10. A three-dimensional coordinate measurement device based on pattern tracking, characterized in that: comprising a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the steps of the three-dimensional coordinate measurement method based on spot tracking described in any one of claims 1 to 4 when executing the program stored in the memory.

Images