Three-dimensional coordinate measuring and calculating method and system based on pattern spot tracking

Abstract

The invention relates to a three-dimensional coordinate measuring and calculating method and system based on pattern spot tracking, and the method comprises the steps: selecting an initial image from a pre-obtained epipolar geometric constraint image pair, extracting an initial pattern spot of a ground feature element, and calculating an initial deviation value between the initial pattern spot and the mass center of a corresponding tracking pattern spot in the image pair; carrying out secondary calibration on the angular points selected in the initial image and the coordinates of the matching points corresponding to the angular points; and calculating the three-dimensional coordinates of all angular points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground feature elements. The method comprises the following steps: extracting an initial pattern spot of an initial image, calculating an initial deviation value between the initial pattern spot and a mass center of a corresponding tracking pattern spot in an image pair, carrying out secondary calibration on pre-selected angular points in the initial image and coordinates of matching points of the angular points, and finally calculating three-dimensional coordinates of all the angular points to obtain three-dimensional coordinates of ground feature elements. 3D eyes and other devices are not needed, the convenience of three-dimensional coordinate measurement of ground feature elements is greatly improved, and the measurement precision is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of photogrammetry, and in particular to a method and system for calculating three-dimensional coordinates based on patch tracking. Background Art

[0002] In the photogrammetry of ground object elements, it is usually necessary to measure the three-dimensional coordinates of the ground object elements. In order to obtain the elevation value in the three-dimensional coordinates, the surveyor needs to wear 3D glasses to measure the elevation value by adjusting the depth of field in the stereoscopic view formed by the image pair. During the measurement process, professional instrument equipment needs to be worn, and certain requirements are imposed on the professional ability of the surveyor. Even so, due to the different observation perspectives, professional abilities, image perception abilities, experience and judgment of different surveyors, there are often certain deviations in the measurement results, which has a great impact on the measurement of the three-dimensional coordinates of ground object elements in photogrammetry. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method and system for calculating three-dimensional coordinates based on patch tracking in view of the above-mentioned deficiencies of the prior art.

[0004] The technical solution of the present invention for solving the above technical problem is as follows: A method for calculating three-dimensional coordinates based on patch tracking includes the following steps: Select an initial image from the pre-acquired image pair with epipolar geometric constraints, extract the initial patch of the ground object element according to the initial image, and calculate the initial offset value between the centroid of the initial patch and the corresponding tracking patch in its image pair; Based on the initial offset value, perform secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image; Based on the calibrated point pairs and the left and right camera parameters, calculate the three-dimensional coordinates of all corner points to obtain the three-dimensional coordinates of the ground object element.

[0005] The beneficial effect of the present invention is that: The method for calculating three-dimensional coordinates based on patch tracking of the present invention extracts the initial patch of the selected initial image in the image pair, calculates the initial offset value between the centroid of the initial patch and the corresponding tracking patch in its image pair, then performs secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value, and finally calculates the three-dimensional coordinates of all corner points based on the calibrated points and the left and right camera parameters to obtain the three-dimensional coordinates of the ground object element. It does not need to rely on devices such as 3D glasses, greatly improves the convenience of measuring the three-dimensional coordinates of ground object elements, overcomes the deviation in manual measurement, and ensures the measurement accuracy.

[0006] On the basis of the above technical solution, the present invention can also be improved as follows: Further: The specific steps for calculating the initial offset value between the centroid of the initial patch and the corresponding patch in its image pair are as follows: Take the initial patch as the target patch to be tracked for the corresponding ground feature element, and use the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image, and obtain the corresponding tracked patch in the image pair of the initial patch; Calculate the abscissa offset value between the initial patch and the tracked patch, and use it as the initial offset value.

[0007] The beneficial effect of the above further solution is that by taking the initial patch as the target patch to be tracked for the corresponding ground feature element, and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracked patch can be accurately obtained, which is convenient for subsequent determination of the initial offset value between the initial patch and the tracked patch.

[0008] Further: The specific steps for using the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image are as follows: Use the image encoding layer composed of a convolutional neural network to perform image feature extraction processing on the image pair with epipolar geometric constraints respectively, and obtain the image features of the image pair; Use the prompt encoding layer to encode the prompts of the points or masks in the initial image in the image pair, and obtain the prompt features; Input the image features and prompt features into the Transformer layer for attention interaction and fusion processing, and transmit them to the mask decoding layer for decoding, and obtain the tracked patch corresponding to the initial patch in the image pair.

[0009] The beneficial effect of the above further solution is that by performing image feature extraction processing on the image pair respectively to obtain the image features of the image pair, and then performing attention interaction and fusion processing in combination with the prompt features extracted from the initial image, and finally decoding to obtain the tracked patch corresponding to the initial patch, so as to accurately determine the initial offset value between the centroid of the initial patch and the corresponding tracked patch in its image pair.

[0010] Further: The specific steps for performing secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value are as follows: Construct a first point set P1 based on the pre-selected corner points in the initial image, and use the initial offset value to perform offset processing on all points in the first point set P1 in the X-axis direction, and construct a second point set P2 based on the points obtained after the offset processing; the calculation formula for the offset processing is: ; ; Among them, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; Taking the coordinates of the points in the second point set P2 as the initial coordinates, searching for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction, and constructing a third point set P3 based on the matching points; Calculating the three-dimensional coordinates of all corner points by using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature elements.

[0011] The beneficial effect of the above further solution is: offsetting the first point set P1 constructed by the corner points in the initial image through the initial offset value, then matching the points in the obtained second point set P2 with the corresponding points in the first point set P1, and finally calculating the three-dimensional coordinates of all corner points based on the obtained third point set P3 and the first point set P1 in combination with the parameters of the left and right cameras, accurately determining the three-dimensional coordinates of the ground feature elements, and realizing the measurement of the three-dimensional coordinates of the ground feature elements.

[0012] Further: The specific steps of searching for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction are as follows: Traverse the points in the first point set P1 , and select an image window with the point as the center in the initial image; among them, the eigenvalue of the point in the first point set P1 feat 1 is the pixel value of the image window. Taking the point after offset processing corresponding to the point in the second point set P2 as the initial value, traversing all points within the preset range in its X-axis direction, and determining the eigenvalue 2 in the second point set P2 and the eigenvalue feat of the point The point with the most similar eigenvalue feat 1 as the matching point of the point in the first point set P1; among them, the eigenvalue of the point in the second point set P2 is the pixel value of the image window centered on this point.

[0013] The beneficial effect of the above further solution is: by selecting an image window with the point in the first point set P1 as the center in the initial image, and then according to the eigenvalue feat 2 of the point after offset processing in the second point set P2 and the eigenvalue featDetermine the matching points of the points in the first point set P1 based on the similarity between 1, thereby facilitating the subsequent calculation of the three-dimensional coordinates of the corner points.

[0014] Further: The eigenvalue in the second point set P2 is determined feat 2 and the point The eigenvalue of feat The specific steps of the point with the most similar eigenvalue 1 include: Calculate the eigenvalue feat 1 and the eigenvalue feat Between 2 gssim Value, the calculation formula is: ; Among them, Represents the average brightness of the corresponding point of the eigenvalue feat 1 in the initial image, Represents the average brightness of the corresponding point of the eigenvalue feat 2 in the initial image, Represents the brightness variance between the corresponding points of the initial image eigenvalue feat 1 and the eigenvalue feat 2, C 1 and C 2 are constants, Is the gradient vector of the corresponding point of the eigenvalue feat 1 in the initial image, Is the gradient vector of the corresponding point of the eigenvalue feat 2 in the initial image; Select the eigenvalue feat 1 with the largest gssim Value of the eigenvalue feat 2 corresponding point as the point corresponding to the point in the first point set P1 The most similar point.

[0015] The beneficial effect of the above further solution is: By calculating the eigenvalue feat 1 and the eigenvalue feat Between 2 gssim Value to determine the eigenvalue of the corresponding point after offset processing in the second point set P2 feat 2 and the eigenvalue of the point in the first point set P1 feat 1 similarity, thus facilitating the determination of the best matching point.

[0016] The present invention also provides a three-dimensional coordinate calculation system based on patch tracking, including an extraction and calculation module, a calibration module, and a three-dimensional calculation module; The extraction and calculation module is used to select an initial image from a pair of images with epipolar geometric constraints obtained in advance, extract the initial patches of ground features according to the initial image, and calculate the initial offset value between the centroid of the initial patch and the corresponding tracking patch in its image pair; The calibration module is used to perform secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value; The three-dimensional calculation module is used to calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters, and obtain the three-dimensional coordinates of the ground feature elements.

[0017] Based on the above technical solutions, the present invention can also be improved as follows: Further: The specific implementation of the extraction and calculation module for calculating the initial offset value between the centroid of the initial patch and the corresponding patch in its image pair is as follows: Take the initial patch as the target patch to be tracked for the corresponding ground feature element, and use the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image, and obtain the corresponding tracked patch in the image pair of the initial patch; Calculate the horizontal coordinate offset value between the initial patch and the tracked patch, and use it as the initial offset value.

[0018] The beneficial effect of the above further solution is that by taking the initial patch as the target patch to be tracked for the corresponding ground feature element, and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracked patch can be accurately obtained, which is convenient for subsequent determination of the initial offset value between the initial patch and the tracked patch.

[0019] Further: The specific implementation of the extraction and calculation module using the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image is as follows: Use the image encoding layer composed of a convolutional neural network to perform image feature extraction processing on the epipolar geometry-constrained image pair respectively to obtain the image features of the image pair; Use the prompt encoding layer to encode the prompts of the points or masks in the initial image in the image pair to obtain prompt features; Input the image features and prompt features into the Transformer layer for attention interaction and fusion processing, and transmit them to the mask decoding layer for decoding to obtain the tracked patch corresponding to the initial patch in the image pair.

[0020] The beneficial effect of the above further solution is that by performing image feature extraction processing on the image pair respectively to obtain the image features of the image pair, and then performing attention interaction and fusion processing in combination with the prompt features extracted from the initial image, and finally decoding to obtain the tracked patch corresponding to the initial patch, so as to accurately determine the initial offset value between the centroid of the initial patch and the corresponding tracked patch in its image pair.

[0021] Further: The specific implementation of the calibration module for performing secondary calibration on the coordinates of the pre-selected corner points and their corresponding matching points in the initial image based on the initial offset value is as follows: Construct a first point set P1 based on the pre-selected corner points in the initial image, and perform offset processing on all points in the first point set P1 in the X-axis direction using the initial offset value. Construct a second point set P2 based on the points obtained after the offset processing. The calculation formula for the offset processing is: ; ; where, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; Use the coordinates of the points in the second point set P2 as the initial coordinates, and search for the matching points corresponding to the points in the first point set P1 within the preset range in the X-axis direction. Construct a third point set P3 based on the matching points; Calculate the three-dimensional coordinates of all corner points using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature elements.

[0022] The beneficial effect of the above further solution is that: the first point set P1 constructed by the corner points in the initial image is offset processed using the initial offset value, then the points in the obtained second point set P2 are matched with the corresponding points in the first point set P1, and finally, based on the obtained third point set P3 and the first point set P1, combined with the parameters of the left and right cameras, the three-dimensional coordinates of all corner points are calculated to accurately determine the three-dimensional coordinates of the ground feature elements and achieve the measurement of the three-dimensional coordinates of the ground feature elements.

[0023] Further: The specific implementation of the calibration module for searching for the matching points corresponding to the points in the first point set P1 within the preset range in the X-axis direction is as follows: Traverse the points in the first point set P1 , and select an image window centered on the point in the initial image; where the eigenvalue of the point feat 1 in the first point set P1 is the pixel value of the image window; Use the point in the second point set P2 corresponding to the offset processed point as the initial value, traverse all points within the preset range in its X-axis direction, and determine the eigenvalue feat 2 in the second point set P2 and the point Eigenvalue feat The point corresponding to the most similar eigenvalue 1 is used as the point in the first point set P1 The matching point; among them, the eigenvalue of the point in the second point set P2 is the pixel value of the image window centered on this point

[0024] The beneficial effect of the above further solution is that by selecting an image window centered on the points in the first point set P1 in the initial image, and then according to the eigenvalue of the corresponding point after offset processing in the second point set P2 feat 2 and the eigenvalue of the point in the first point set P1 feat 1 to determine the matching point of the point in the first point set P1, which is convenient for subsequent calculation of the three-dimensional coordinates of the corner point

[0025] Further: The calibration module determines the eigenvalue in the second point set P2 feat 2 and the point The specific implementation of the most similar eigenvalue of feat 1 is as follows Calculate the eigenvalue feat 1 and the eigenvalue feat 2 between the gssim value, and the calculation formula is ; Among them, Represents the average brightness of the point corresponding to the eigenvalue feat 1 in the initial image Represents the average brightness of the point corresponding to the eigenvalue feat 2 in the initial image Represents the initial image eigenvalue feat 1 and eigenvalue feat 2 the brightness variance between the corresponding points C 1 and C 2 are constants Is the gradient vector of the point corresponding to the eigenvalue in the initial image feat 1 Is the gradient vector of the point corresponding to the eigenvalue in the initial image feat 2; Select the eigenvalue with the largest feat 1 gssim value as the eigenvalue feat 2 that is most similar to the point in the first point set P1

[0026] The beneficial effect of the above further solution is that by calculating the eigenvalue feat 1 and the eigenvalue feat 2 between the gssim value to determine the eigenvalue of the corresponding point after offset processing in the second point set P2 feat ​The similarity between 2 and the eigenvalue of the points in the first point set P1, so as to facilitate the determination of the best matching points. feat between 1, thus facilitating the determination of the best matching points.

[0027] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the three-dimensional coordinate calculation method based on patch tracking.

[0028] The present invention also provides a three-dimensional coordinate calculation device based on patch tracking, including a communication interface, a memory, a communication bus, and a processor, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is used to implement the steps of the three-dimensional coordinate calculation method based on patch tracking when executing the program stored on the memory. Description of the Drawings

[0029] Figure 1 is a schematic flowchart of a three-dimensional coordinate calculation method based on patch tracking according to an embodiment of the present invention; Figure 2a is a schematic example diagram of the left image in an image pair according to an embodiment of the present invention; Figure 2b is a schematic example diagram of the right image in an image pair according to an embodiment of the present invention; Figure 3 is a schematic example diagram of the initial patch of a ground feature according to an embodiment of the present invention; Figure 4 is a schematic example diagram of the tracked patch of a ground feature according to an embodiment of the present invention; Figure 5 is a schematic diagram of the matching result of the pre-selected corner points and their corresponding matching points in the initial image of a ground feature according to an embodiment of the present invention; Figure 6 is a schematic flowchart of a three-dimensional coordinate calculation method based on patch tracking according to an embodiment of the present invention. Detailed Embodiments

[0030] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0031] As Figure 1 shown, a three-dimensional coordinate calculation method based on patch tracking includes the following steps: S1: Select an initial image from a pre-acquired image pair with epipolar geometric constraints, extract the initial patch of the ground feature according to the initial image, and calculate the initial offset value between the centroid of the initial patch and the corresponding tracked patch in its image pair; S2: Based on the initial offset value, perform secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image; S3: Calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground feature elements.

[0032] The three-dimensional coordinate measurement method based on patch tracking of the present invention extracts the initial patch of the selected initial image in the image pair, calculates the initial offset value between the centroid of the initial patch and the corresponding tracked patch in its image pair, then performs secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value, and finally calculates the three-dimensional coordinates of all corner points based on the calibrated points and the left and right camera parameters to obtain the three-dimensional coordinates of the ground feature elements. Without the need for devices such as 3D glasses, it greatly improves the convenience of measuring the three-dimensional coordinates of ground feature elements, overcomes the deviation in manual measurement, and ensures the measurement accuracy.

[0033] In one or more embodiments of the present invention, the specific steps for calculating the initial offset value between the centroid of the initial patch and the corresponding patch in its image pair are as follows: S11: Use the initial patch as the target patch to be tracked for the corresponding ground feature element, and use the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image to obtain the corresponding tracked patch in the image pair of the initial patch; S12: Calculate the horizontal coordinate offset value between the initial patch and the tracked patch and use it as the initial offset value.

[0034] By using the initial patch as the target patch to be tracked for the corresponding ground feature element and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracked patch can be accurately obtained, which is convenient for subsequent determination of the initial offset value between the initial patch and the tracked patch.

[0035] In one or more embodiments of the present invention, in the image pair with epipolar geometric constraints, the left image is selected as the initial image, and the initial patch of the ground feature element is extracted on the left image. The image pair, that is, the left and right images, is regarded as a 2-frame video, where the left image is the first frame, and the initial patch of the ground feature element is the target patch to be tracked, and then the epipolar image pair tracking model is used to process it to obtain the patch of the ground feature element in the right image.

[0036] In the embodiments of the present invention, Figure 2a - Figure 2b Taking the shown image pair as an example, a detailed description of three-dimensional coordinate measurement is given.

[0037] Taking the left image as the initial image, manually measure a certain ground feature element to obtain the initial patch of the ground feature element, such asFigure 3 as shown in the red area

[0038] Specifically, in one or more embodiments of the present invention, the tracking process of the target patch in the image pair of the initial image by using the epipolar image pair tracking model specifically includes the following steps: S111: Use the image encoding layer composed of a convolutional neural network to perform image feature extraction processing on the image pairs with epipolar geometric constraints respectively to obtain the image features of the image pairs; S112: Use the prompt encoding layer to encode the prompts of the points or masks in the initial image in the image pair to obtain prompt features; S113: Input the image features and prompt features into the Transformer layer for attention interaction and fusion processing, and transmit them to the mask decoding layer for decoding to obtain the tracking patch corresponding to the initial patch in the image pair.

[0039] By performing image feature extraction processing on the image pairs respectively to obtain the image features of the image pairs, and then performing attention interaction and fusion processing in combination with the prompt features extracted from the initial image, and finally decoding to obtain the tracking patch corresponding to the initial patch, the initial offset value between the centroid of the initial patch and the corresponding tracking patch in its image pair can be accurately determined.

[0040] In practice, the left image in the image pair (such as Figure 2a shown) is used as the first frame, and the right image in the image pair (such as Figure 2b shown) is used as the second frame. Figure 3 The feature map patch of the ground object in is used as the initial marked patch of the first frame and input into the epipolar image pair tracking model, and the patch of the initial marked patch in the second frame, that is, the right image, is output, as shown in Figure 4 the red area

[0041] In one or more embodiments of the present invention, when calculating the abscissa offset value between the initial patch and the tracking patch, according to the epipolar geometric constraint, the corresponding points of the image pair are conjugate and the ordinates are the same, so only the abscissa offset value of the centroid is taken as the initial offset value.

[0042] In one or more embodiments of the present invention, the secondary calibration of the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value specifically includes the following steps: S21: Construct a first point set P1 based on the pre-selected corner points in the initial image, and use the initial offset value to perform offset processing on all points in the first point set P1 in the X-axis direction, and construct a second point set P2 based on the points obtained after the offset processing; the calculation formula for the offset processing is: ; ; Among them, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; In practice, calculate the two-dimensional centroid coordinates of the patch in the left image ( x 1 ,y 1), and the two-dimensional centroid coordinates of the patch in the right image ( x 2 ,y 2). Since the left and right images satisfy the epipolar geometry constraint, the corresponding points in the image pair only have x axis, that is, the offset in the abscissa direction. Therefore, only calculate the offset value of the abscissa offset ( x ) =x 1 -x 2.

[0043] Specifically, take all the corner point coordinates manually extracted in the left image as a point set P1, and the number of corner points is n , for these n points, add an offset in the x axis direction offset ( x ) to form a new point set P2.

[0044] S22: Use the coordinates of the points in the second point set P2 as the initial coordinates, and search for the matching points corresponding to the points in the first point set P1 within the preset range in the X-axis direction, and construct a third point set P3 based on the matching points; S23: Calculate the three-dimensional coordinates of all corner points using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to obtain the three-dimensional coordinates of the ground feature.

[0045] Perform offset processing on the first point set P1 constructed by the corner points in the initial image through the initial offset value, then match the points in the obtained second point set P2 with the corresponding points in the first point set P1, and finally calculate the three-dimensional coordinates of all corner points based on the obtained third point set P3 and the first point set P1 combined with the parameters of the left and right cameras, accurately determine the three-dimensional coordinates of the ground feature, and realize the measurement of the three-dimensional coordinates of the ground feature.

[0046] In one or more embodiments of the present invention, the searching for the matching points corresponding to the points in the first point set P1 within the preset range in the X-axis direction specifically includes the following steps: S221: Traverse the points in the first point set P1 , select an image window centered at a point in the initial image ; among which, the eigenvalue of the point in the first point set P1 feat 1 is the pixel value of the image window S222: Using the point corresponding to the offset processing in the second point set P2 as the initial value, traverse all points within the preset range in its X-axis direction, and determine the eigenvalue 2 in the second point set P2 and the eigenvalue feat of the point 1 is the most similar point as the matching point of the point feat 1 in the first point set P1; among which, the eigenvalue of the point in the second point set P2 is the pixel value of the image window centered at this point. Specifically, traverse the points in P1, denoted as

[0047] , take an 11x11 image window centered at this point in the left image, and the pixel value of this window is used as the feature of this point, denoted as 1. At the same time, use the point feat corresponding to this point in P2 as the initial value, and traverse all points within the range of [-5,5] in its axis direction. The feature of each point is also the pixel value of an 11x11 window centered at this point. There are 11 points in this range, so there are 11 eigenvalue. By comparing with x 1, the eigenvalue most similar to feat 1 is obtained, and the point where this eigenvalue is located is denoted as the matching point after secondary calibration and recorded in P3. feat 1.

[0048] By selecting an image window centered at the points in the first point set P1 in the initial image, and then determining the matching points of the points in the first point set P1 according to the similarity between the eigenvalue feat 2 of the corresponding offset points in the second point set P2 and the eigenvalue feat 1 of the points in the first point set P1, it is convenient to calculate the three-dimensional coordinates of the corner points in the subsequent calculation.

[0049] In one or more embodiments of the present invention, the specific steps of determining the point with the eigenvalue feat 2 in the second point set P2 that is most similar to the eigenvalue of the point feat 1 include: S2221: Calculate the feat value between the eigenvalue feat 1 and the eigenvalue gssim 2, and the calculation formula is: ; Among them, represents the average brightness of the corresponding points of the eigenvalue feat 1 in the initial image, represents the average brightness of the corresponding points of the eigenvalue feat 2 in the initial image, represents the brightness variance between the corresponding points of the eigenvalue feat 1 and the eigenvalue feat 2 in the initial image feature, C 1 and C 2 are constants, is the gradient vector of the corresponding points of the eigenvalue feat 1 in the initial image, is the gradient vector of the corresponding points of the eigenvalue feat 2 in the initial image; S2222: Select the eigenvalue feat 2 corresponding to the eigenvalue with the largest gssim value of feat 1 as the point most similar to the points in the first point set P1 .

[0050] In the present invention, by calculating the gssim value between two eigenvalues, the largest gssim value is taken. gssim The larger the n value, the more similar the structures are. After corresponding the corner points of the first point set P1 and the third point set P3, Figure 5 the corresponding matching points are obtained, and the matching result is as

[0051] shown. feat By calculating the feat value between the eigenvalue gssim 1 and the eigenvalue feat 2, the similarity between the eigenvalue feat 2 of the corresponding points after offset processing in the second point set P2 and the eigenvalue

[0052] 1 of the points in the first point set P1 is determined, so as to facilitate the determination of the best matching point.

[0053] As Figure 6 shown, the present invention also provides a three-dimensional coordinate calculation system based on patch tracking, including an extraction and calculation module, a calibration module and a three-dimensional calculation module; The extraction and calculation module is used to select an initial image from the pre-acquired image pair with epipolar geometric constraints, extract the initial image spot of the ground feature element according to the initial image, and calculate the initial offset value between the initial image spot and the centroid of the corresponding tracking image spot in the image pair; The calibration module is used to perform secondary calibration on the pre-selected corner points in the initial image and the corresponding matching point coordinates based on the initial offset values; The three-dimensional calculation module is used to calculate the three-dimensional coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters to obtain the three-dimensional coordinates of the ground features.

[0054] In one or more embodiments of the present invention, the specific implementation of the extraction calculation module calculating the initial offset value between the initial spot and the centroid of the corresponding spot in the image pair is: The initial image spot is used as the target image spot that needs to be tracked for the corresponding ground feature, and the target image spot is tracked in the image pair of the initial image using the epipolar line image pair tracking model to obtain the corresponding tracking image spot in the image pair of the initial image spot; The horizontal coordinate offset value between the initial image spot and the tracking image spot is calculated and used as the initial offset value.

[0055] By taking the initial spot as the target spot that needs to be tracked for the corresponding ground feature, and then using the epipolar image pair tracking model to perform tracking processing in the image pair of the initial image, the tracking spot can be accurately obtained, which makes it convenient to subsequently determine the initial offset value between the initial spot and the tracking spot.

[0056] In one or more embodiments of the present invention, the specific implementation of the extraction calculation module tracking the target spot in the image pair of the initial image using the epipolar line image pair tracking model is as follows: The image encoding layer composed of convolutional neural networks is used to extract image features from the image pairs with kernel line geometric constraints to obtain the image features of the image pairs. The prompt coding layer is used to encode the prompt of the point or mask in the initial image of the image pair to obtain the prompt feature; The image features and prompt features are input into the Transformer layer for attention interaction fusion processing, and transmitted to the mask decoding layer for decoding to obtain the tracking spot corresponding to the initial spot in the image pair.

[0057] The image features of the image pair are obtained by performing image feature extraction processing on the image pair respectively, and the attention cross-line fusion processing is performed on the prompt features extracted from the initial image. Finally, the tracking spot corresponding to the initial spot is obtained by decoding, so as to accurately determine the initial offset value between the initial spot and the center of mass of the corresponding tracking spot in the image pair.

[0058] In one or more embodiments of the present invention, the specific implementation of the calibration module for performing secondary calibration on the preselected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value is as follows: Construct a first point set P1 based on the preselected corner points in the initial image, and perform offset processing on all points in the first point set P1 in the X-axis direction by using the initial offset value. Construct a second point set P2 based on the points obtained after the offset processing. The calculation formula for the offset processing is: ; ; where, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; Use the coordinates of the points in the second point set P2 as the initial coordinates, and search for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction. Construct a third point set P3 based on the matching points; Calculate the three-dimensional coordinates of all corner points by using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras, and obtain the three-dimensional coordinates of the ground feature elements.

[0059] Perform offset processing on the first point set P1 constructed by the corner points in the initial image through the initial offset value, then match the points in the obtained second point set P2 with the corresponding points in the first point set P1, and finally calculate the three-dimensional coordinates of all corner points based on the obtained third point set P3 and the first point set P1 in combination with the parameters of the left and right cameras, accurately determine the three-dimensional coordinates of the ground feature elements, and realize the measurement of the three-dimensional coordinates of the ground feature elements.

[0060] In one or more embodiments of the present invention, the specific implementation of the calibration module for searching for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction is as follows: Traverse the points in the first point set P1 , and select an image window in the initial image with the point as the center; where the eigenvalue of the point feat 1 in the first point set P1 is the pixel value of the image window; Use the point in the second point set P2 corresponding to the offset processed point as the initial value, traverse all points within the preset range in its X-axis direction, and determine the eigenvalue in the second point set P2feat The point with the eigenvalue most similar to 2 is used as the matching point of the point in the first point set P1; where the eigenvalue of the point in the second point set P2 is the pixel value of the image window centered on this point. feat The matching point of the point in the first point set P1 is determined by selecting an image window centered on the point in the first point set P1 in the initial image, and then according to the eigenvalue of the corresponding point after offset processing in the second point set P2 and the similarity between the eigenvalue of the point in the first point set P1. This facilitates the subsequent calculation of the three-dimensional coordinates of the corner points.

[0061] In one or more embodiments of the present invention, the specific implementation of the calibration module to determine the point with the eigenvalue feat 2 most similar to the eigenvalue of the point feat 1 is as follows:

[0062] Calculate the feat value between the eigenvalue 1 and the eigenvalue feat 2. The calculation formula is: Calculate the feat value between the eigenvalue feat 1 and the eigenvalue gssim 2. The calculation formula is: ; Where, represents the average brightness of the corresponding point of the eigenvalue feat 1 in the initial image, represents the average brightness of the corresponding point of the eigenvalue feat 2 in the initial image, represents the brightness variance between the corresponding points of the initial image eigenvalue feat 1 and the eigenvalue feat 2, C 1 and C 2 are constants, is the gradient vector of the corresponding point of the eigenvalue feat 1 in the initial image, is the gradient vector of the corresponding point of the eigenvalue feat 2 in the initial image; Select the eigenvalue feat 2 corresponding to the point with the largest gssim value of the eigenvalue feat 1 as the point most similar to the point in the first point set P1 .

[0063] The eigenvalue of the corresponding point after offset processing in the second point set P2 is determined by calculating the feat value between the eigenvalue feat 1 and the eigenvalue gssim 2.feat The similarity between the eigenvalues of the points in the first point set P1 and 2 feat 1, so as to facilitate the determination of the best matching points.

[0064] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional coordinate calculation method based on patch tracking as described above.

[0065] The present invention also provides a three-dimensional coordinate calculation device based on patch tracking, including a communication interface, a memory, a communication bus and a processor, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; The processor is used to implement the steps of the three-dimensional coordinate calculation method based on patch tracking when executing the program stored on the memory.

[0066] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional coordinate calculation method based on patch tracking, characterized in that, It includes the following steps: Select an initial image from a pre-acquired image pair with epipolar geometry constraints, extract an initial patch of a ground feature from the initial image, and calculate an initial offset value between the centroid of the initial patch and the centroid of the corresponding tracking patch in its image pair; Based on the initial offset value, perform secondary calibration on the pre-selected corner points and their corresponding matching point coordinates in the initial image; Based on the calibrated point pairs and the left and right camera parameters, calculate the three-dimensional coordinates of all corner points to obtain the three-dimensional coordinates of the ground feature.

2. The three-dimensional coordinate calculation method based on patch tracking according to claim 1, characterized in that The calculation of the initial offset value between the centroid of the initial patch and the centroid of the corresponding patch in its image pair specifically includes the following steps: Take the initial patch as the target patch to be tracked for the corresponding ground feature, and use the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image to obtain the corresponding tracking patch in the image pair of the initial patch; Calculate the abscissa offset value between the initial patch and the tracking patch, and use it as the initial offset value.

3. The three-dimensional coordinate measurement method based on patch tracking according to claim 2, wherein The use of the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image specifically includes the following steps: Use the image encoding layer composed of a convolutional neural network to perform image feature extraction processing on the image pair with epipolar geometry constraints respectively to obtain the image features of the image pair; Use the prompt encoding layer to encode the prompts of the points or masks in the initial image in the image pair to obtain prompt features; Input the image features and prompt features into the Transformer layer for attention interaction and fusion processing, and transmit them to the mask decoding layer for decoding to obtain the tracking patch corresponding to the initial patch in the image pair.

4. The three-dimensional coordinate measurement method based on patch tracking according to claim 1, characterized in that The secondary calibration of the pre-selected corner points and their corresponding matching point coordinates in the initial image based on the initial offset value specifically includes the following steps: Construct a first point set P1 based on the pre-selected corner points in the initial image, and use the initial offset value to perform offset processing on all points in the first point set P1 in the X-axis direction, and construct a second point set P2 based on the points obtained after the offset processing; the calculation formula for the offset processing is: ； offset ( x )= x 1- x 2； Among them, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; Take the coordinates of the points in the second point set P2 as the initial coordinates, and search for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction, and construct a third point set P3 based on the matching points; Use the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras to calculate the three-dimensional coordinates of all corner points to obtain the three-dimensional coordinates of the ground feature.

5. The three-dimensional coordinate measurement method based on patch tracking according to claim 4, characterized in that, The search for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction specifically includes the following steps: Traverse the points in the first point set P1 , and select an image window centered on the point in the initial image; among them, the eigenvalue of the point in the first point set P1 feat 1 is the pixel value of the image window; Using the point after corresponding offset processing in the second point set P2 as the initial value, traverse all points within the preset range in its X-axis direction, and determine the eigenvalue 2 in the second point set P2, and the point with the eigenvalue feat closest to the eigenvalue 1 of the point feat as the matching point of the point in the first point set P1; where the eigenvalue of a point in the second point set P2 is the pixel value of the image window centered on that point; The specific steps for determining the point in the second point set P2 whose eigenvalue feat 2 is most similar to the eigenvalue of the point feat 1 are as follows: Calculate the eigenvalue feat between 1 and the eigenvalue feat 2 is gssim calculated by the formula: ； Among them, represents the average brightness of the corresponding point of eigenvalue feat 1 in the initial image, represents the average brightness of the corresponding point of eigenvalue feat 2 in the initial image, represents the brightness variance between the corresponding points of eigenvalue feat 1 and eigenvalue feat 2 in the initial image, C 1 and C 2 are constants, is the gradient vector of the corresponding point of eigenvalue feat 1 in the initial image, is the gradient vector of the corresponding point of eigenvalue feat 2 in the initial image; Select the eigenvalue corresponding to feat 1 gssim with the largest value feat 2, and use the corresponding point as the point most similar to the points in the first point set P1.

6. A three-dimensional coordinate measurement system based on patch tracking, characterized in that, It includes an extraction and calculation module, a calibration module, and a three-dimensional calculation module; The extraction and calculation module is used to select an initial image from a pre-acquired image pair with epipolar geometric constraints, extract an initial patch of a ground feature from the initial image, and calculate an initial offset value between the centroid of the initial patch and the centroid of the corresponding traced patch in its image pair; The calibration module is used to perform secondary calibration on the coordinates of pre-selected corner points and their corresponding matching points in the initial image based on the initial offset value; The 3D calculation module is used to calculate the 3D coordinates of all corner points based on the calibrated point pairs and the left and right camera parameters, and obtain the 3D coordinates of the ground feature.

7. The three-dimensional coordinate measurement system based on patch tracking according to claim 6, characterized in that The specific implementation of the extraction and calculation module for calculating the initial offset value between the centroid of the initial patch and the centroid of the corresponding patch in its image pair is as follows: Take the initial patch as the target patch to be traced for the corresponding ground feature, and use the epipolar image pair tracking model to perform tracking processing on the target patch in the image pair of the initial image, and obtain the corresponding traced patch in the image pair of the initial patch; Calculate the abscissa offset value between the initial patch and the traced patch, and use it as the initial offset value; The specific implementation of the extraction and calculation module for performing tracking processing on the target patch in the image pair of the initial image using the epipolar image pair tracking model is as follows: Use the image encoding layer composed of a convolutional neural network to perform image feature extraction processing on the image pair with epipolar geometric constraints respectively, and obtain the image features of the image pair; Use the prompt encoding layer to encode the prompts of points or masks in the initial image in the image pair, and obtain prompt features; Input the image features and prompt features into the Transformer layer for attention interaction and fusion processing, and transmit them to the mask decoding layer for decoding, and obtain the traced patch corresponding to the initial patch in the image pair.

8. The 3D coordinate measurement system based on patch tracking according to claim 6, wherein The specific implementation of the calibration module for performing secondary calibration on the coordinates of pre-selected corner points and their corresponding matching points in the initial image based on the initial offset value is as follows: Construct a first point set P1 based on the pre-selected corner points in the initial image, and use the initial offset value to perform offset processing on all points in the first point set P1 in the X-axis direction, and construct a second point set P2 based on the points obtained after the offset processing; the formula for the offset processing is: ； ； Among them, x 1 is the abscissa of the two-dimensional centroid of the initial patch in the initial image, x 2 is the abscissa of the two-dimensional centroid of the tracked patch in the image pair of the initial image, offset ( x ) is the initial offset value; Use the coordinates of the points in the second point set P2 as the initial coordinates, and search for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction, and construct a third point set P3 based on the matching points; Calculate the 3D coordinates of all corner points using the matching points of the first point set P1 and the third point set P3 and the parameters of the left and right cameras, and obtain the 3D coordinates of the ground feature; The specific implementation of the calibration module for searching for matching points corresponding to the points in the first point set P1 within a preset range in the X-axis direction is as follows: Traverse the points in the first point set P1 , and select an image window centered at the point in the initial image; among them, the eigenvalue of the point in the first point set P1 feat 1 is the pixel value of the image window; Using the point after corresponding offset processing in the second point set P2 as the initial value, traverse all points within a preset range in its X-axis direction, and determine the eigenvalue in the second point set P2. The point corresponding to the eigenvalue feat 2 that is most similar to the eigenvalue of the point feat 1 is used as the matching point of the point in the first point set P1; where the eigenvalue of a point in the second point set P2 is the pixel value of the image window centered on that point; The calibration module determines the eigenvalue in the second point set P2 feat 2 and the point eigenvalue of feat 1. The specific implementation of the most similar point is as follows: Calculate the eigenvalue feat between 1 and the eigenvalue feat 2 is gssim calculated by the formula: ； Among them, represents the average brightness of the corresponding points of eigenvalue feat 1 in the initial image, represents the average brightness of the corresponding points of eigenvalue feat 2 in the initial image, represents the brightness variance between the corresponding points of eigenvalue feat 1 and eigenvalue feat 2 in the initial image, C 1 and C 2 are constants, is the gradient vector of the corresponding points of eigenvalue feat 1 in the initial image, is the gradient vector of the corresponding points of eigenvalue feat 2 in the initial image; Select the eigenvalue corresponding to feat 1 gssim with the largest value feat 2 as the point that is most similar to the points in the first point set P1.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method for calculating three-dimensional coordinates based on patch tracking according to any one of claims 1-5.

10. A three-dimensional coordinate measurement device based on patch tracking, characterized in that: It includes a communication interface, a memory, a communication bus, and a processor. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus; The memory is used to store a computer program; When the processor is used to execute the program stored on the memory, it implements the steps of the method for calculating three-dimensional coordinates based on patch tracking according to any one of claims 1 to 5.

Images