Visual layout method based on image matching and GNSS integrated navigation system

Abstract

The present invention discloses a visual lofting method based on image matching and a GNSS integrated navigation system. The GNSS integrated navigation system includes a downward-looking lens for shooting downwards. The visual lofting method includes: obtaining three-dimensional point cloud data of a target area; obtaining a bird's-eye view of the target area using the three-dimensional point cloud data of the target area; obtaining position and posture data of the GNSS integrated navigation system near the target area and image data of the target area; performing image matching on pixels of landmark points in the bird's-eye view and pixels in the image data; obtaining a deviation correction amount between the actual position and posture of the downward-looking lens and the position and posture estimated by the GNSS integrated navigation system using image matching; obtaining the coordinates of the landmark points in the image data using the correction amount of the position and posture data of the GNSS integrated navigation system. The present invention can improve the lofting accuracy near the obstruction, and improve the lofting accuracy of the equipment from the decimeter level to the centimeter level.

Description

Technical Field

[0001] The present invention relates to the field of GNSS integrated navigation systems, and in particular to a visual lofting method based on image matching and a GNSS integrated navigation system. Background Art

[0002] GNSS-based engineering surveying technology has developed over decades, from centering measurement to the tilt measurement that is widely used today. In recent years, the release of GNSS integrated navigation systems has provided a new direction for engineering surveying technology.

[0003] Staking is an important technology in surveying and mapping projects. This technology displays known points on the map, guiding users to accurately find and poke known points to achieve accurate engineering construction. Generally speaking, for high-precision surveying and mapping engineering needs, the stakeout accuracy should reach about 2cm. Visual stakeout is to install a camera in the device to display the above-mentioned known points in the video, and through the AR real scene, it makes it easier for users to find stakeout points. This technology provides convenience for stakeout work.

[0004] Both traditional and visual stakeout rely on GNSS to provide accurate location information, which is the basis for the known points to be accurately displayed in the above maps or videos. In other words, the basis for the stakeout accuracy to reach 2 cm is that the GNSS measurement accuracy can be stabilized at around 2 cm. However, in obstructed environments, such as corners and shade, GNSS, and even the currently commonly used GNSS / INS combined navigation system, will produce more or less position offsets due to the inherent defects of their technology, and the error is difficult to guarantee an accuracy of 2 cm. On the contrary, the error can even reach the decimeter level. In this case, it is difficult to accurately display the known points, and accurate stakeout cannot be achieved.

[0005] Generally speaking, the key to improving the above-mentioned layout accuracy lies in improving the positioning accuracy. However, in the above-mentioned environment, it is really difficult to improve the positioning accuracy. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the defect in the prior art that the GNSS integrated navigation system is difficult to improve the positioning accuracy in an environment near obstructions, and to provide a visual layout method and a GNSS integrated navigation system based on image matching that can improve the layout accuracy near obstructions and improve the decimeter-level layout accuracy of the equipment to the centimeter-level.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] A visual lofting method based on image matching is used in a GNSS integrated navigation system, wherein the GNSS integrated navigation system includes a downward-looking lens for shooting downwards, and is characterized in that the visual lofting method includes:

[0009] Acquire three-dimensional point cloud data of a target area;

[0010] Acquire a top view of the target area using the three-dimensional point cloud data of the target area;

[0011] Acquire position and attitude data of the GNSS integrated navigation system near the target area and image data of the target area;

[0012] For a landmark point in the target area, image matching is performed between pixels of the landmark point in the overhead view and pixels in the image data;

[0013] Using the image matching, the deviation correction amount between the real position and posture of the downward-looking lens and the position and posture estimated by the GNSS integrated navigation system is obtained;

[0014] The coordinates of the landmark points in the image data are obtained by using the correction amount of the position and attitude data of the GNSS integrated navigation system.

[0015] Preferably, the GNSS integrated navigation system includes a downward-looking lens for shooting downward, the image data is image data shot by the downward-looking lens, and the image matching of pixels of the landmark point in the overhead view with pixels in the image data includes:

[0016] Obtain three-dimensional point cloud data of landmark points;

[0017] For a target moment, obtain the position and posture data of the downward-looking lens and the camera projection model;

[0018] The position and posture data of the downward-looking lens and the camera projection model are used to establish a formula for projecting the 3D point cloud data onto the overhead view.

[0019] The expression formula is used to perform image matching between the pixels of the landmark point in the overhead view and the pixels in the image data.

[0020] Preferably, the expression formula is:

[0021] ,

[0022] in, is the position data of the downward-looking lens, is the posture data of the downward-looking lens, is the three-dimensional point cloud data of landmark points. is the camera projection model, is the position of the landmark point in the top view.

[0023] Preferably, the step of obtaining a top view of the target area by using the three-dimensional point cloud data of the target area includes:

[0024] Acquire an initial top view of the target area using the three-dimensional point cloud data of the target area, and perform bilinear interpolation on the initial top view to acquire the top view;

[0025] or,

[0026] The image matching of the pixels of the landmark point in the overhead view with the pixels in the image data comprises:

[0027] Perform bilinear interpolation on the pixels in the area where the landmark points in the top view are located;

[0028] Obtain image matching between the locations of landmark points in the interpolated top view and those in the image data.

[0029] Preferably, the using the expression formula to perform image matching between pixels of the landmark point in the overhead view and pixels in the image data comprises:

[0030] Using the expression formula, a cluster of top-view coordinates of landmark points are obtained;

[0031] A cluster of top view coordinates and a cluster of image data are subjected to the image matching.

[0032] Preferably, the visual lofting method comprises:

[0033] Using the image matching to construct the estimation problem of the position and posture data deviation of the downward-looking lens;

[0034] Minimizing the estimation problem to obtain a bias correction amount;

[0035] The deviation correction amount is used to obtain the real-time coordinates of the landmark points in the image data.

[0036] Preferably, the estimation problem is:

[0037] ,

[0038] in, is the estimated value of the position and attitude deviation of the downward-looking lens, is a cluster of top-view coordinates of landmark points, is a cluster of image data of landmark points, is the luminosity value of the pixel position of the landmark point in the image data, is the inverse transformation of the camera projection model.

[0039] Preferably, using Get the real-time coordinates of landmark points in image data, is the position of the landmark point in the image data, and the visual lofting method includes: performing AR real-scene display of the real-time coordinates.

[0040] Preferably, the GNSS integrated navigation system includes a forward-looking lens, and the visual lofting method includes:

[0041] When moving away from the target area, a plurality of frames of dynamic images of the target area are captured by a front-view camera to obtain position and attitude data of the GNSS integrated navigation system during the process of moving to the target area;

[0042] When approaching the target area, the downward-looking lens is used to obtain image data of the target area, and the position and attitude data of the GNSS integrated navigation system during the process of moving to the target area;

[0043] Acquire three-dimensional point cloud data of the target area using the plurality of frames of dynamic images;

[0044] The present invention also provides a GNSS integrated navigation system, which is characterized in that the GNSS integrated navigation system is used to implement the above-mentioned visual lofting method based on image matching.

[0045] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.

[0046] The positive and progressive effects of the present invention are:

[0047] The present invention can improve the positioning accuracy near the shielding object and improve the positioning accuracy of the equipment from the decimeter level to the centimeter level. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a schematic diagram of the structure of the GNSS integrated navigation system according to Example 1 of the present invention.

[0049] Figure 2 This is a schematic diagram of the working principle of the GNSS integrated navigation system according to Example 1 of the present invention.

[0050] Figure 3 Flow chart of the visual lofting method according to embodiment 1 of the present invention. DETAILED DESCRIPTION

[0051] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0052] Example 1

[0053] This embodiment provides a GNSS integrated navigation system, which includes a GNSS receiver, an INS module and a vision module. In other embodiments, the GNSS integrated navigation system is a receiver including a GNSS module, an INS module and a vision module.

[0054] In this embodiment, the visual module includes a downward-looking lens for shooting downward and a front-looking lens. In other embodiments, the downward-looking lens and the front-looking lens can be a rotatable lens.

[0055] The front-view camera is used to obtain three-dimensional point cloud data of a target area. The three-dimensional point cloud data can be obtained by using a three-dimensional model established by a two-dimensional camera through multiple frames of images, or by using other methods.

[0056] The processing module of the GNSS integrated navigation system is used for:

[0057] Acquire a top view of the target area using the three-dimensional point cloud data of the target area;

[0058] The position and attitude data of the GNSS integrated navigation system near the target area and the image data of the target area are obtained, wherein the position and attitude data include the positioning data obtained by the GNSS receiver or module and the attitude data obtained by the INS module. The image data is an image obtained by the downward-looking lens near the obstruction.

[0059] In this embodiment, the vicinity of the target area refers to a preset range from the target area, such as a range of 3 meters to 5 meters from the target area is considered to be the vicinity of the target area.

[0060] For a landmark point in the target area, image matching is performed between pixels of the landmark point in the overhead view and pixels in the image data;

[0061] The entire cluster of pixels around the target point is matched as a whole. In other words, the entire photo taken by the downward-looking camera is matched with the top view generated by the 3D point cloud.

[0062] Using the image matching, the deviation correction amount between the real position and posture of the downward-looking lens and the position and posture estimated by the GNSS integrated navigation system is obtained;

[0063] The position and attitude estimated by the GNSS integrated navigation system is the position and attitude data estimated by the system.

[0064] The coordinates of the landmark points in the image data are obtained by using the correction amount of the position and attitude data of the GNSS integrated navigation system.

[0065] Get the location of the landmark points in the image data and display the coordinates in real-time AR scene.

[0066] The processing module of the GNSS integrated navigation system is used for:

[0067] Obtain three-dimensional point cloud data of landmark points;

[0068] For a target moment, obtain the position and posture data of the downward-looking lens and the camera projection model;

[0069] The position and posture data of the downward-looking lens and the camera projection model are used to establish a formula for projecting the 3D point cloud data onto the overhead view.

[0070] The expression formula is used to perform image matching between the pixels of the landmark point in the overhead view and the pixels in the image data.

[0071] like Figure 1 As shown, the GNSS integrated navigation system 11 involved in this embodiment is different from the traditional GNSS equipment, and includes a downward-looking lens and a forward-looking lens.

[0072] The front-view lens 111 is used for measurement, and the downward-view lens 112 is used for layout.

[0073] like Figure 2 As shown, the work is divided into two parts. Specifically, in the first part, namely visual measurement, the process is as follows:

[0074] 1-1) The user holds the device away from obstructions 12 to ensure reliable GNSS performance;

[0075] 1-2) The user records a set of videos through certain movements to cover the surrounding environment of the point to be staked out;

[0076] 1-3) The algorithm uses GNSS, inertial navigation, and forward-looking camera information to estimate the pose, and further performs three-dimensional reconstruction to obtain a dense three-dimensional point cloud around the point to be staked out.

[0077] In the second part, namely visual matching lofting, the process is as follows:

[0078] 2-1) The algorithm projects the above dense 3D point cloud into a top view so that the perspective of the downward camera is similar to the perspective of the point cloud;

[0079] 2-2) The user moves the handheld device to the vicinity of the point to be staked out. At this time, since the GNSS is blocked by the obstruction 12, its positioning performance will be reduced. If no further algorithm operation is performed, the display of the point to be staked out on the map or image will be offset;

[0080] 2-3) When the algorithm determines that the user's location is close to the location of the point to be staked out (the error of decimeter to meter level is sufficient for this judgment), the algorithm will start visual matching, that is, match the image of the downward camera with the point cloud projected onto the top view, and use this as a basis to correct the location display of the point to be staked out in the AR real scene image.

[0081] Specifically, the expression formula is:

[0082] ,

[0083] in, is the position data of the downward-looking lens, is the posture data of the downward-looking lens, is the 3D point cloud data of landmark points. is the camera projection model, is the location of the landmark point in the top view.

[0084] In order to facilitate the matching of the downward-looking camera and the point cloud, the present invention will perform projection based on the position, posture, pixels and viewing angle of the downward-looking camera. is a landmark point in the 3D point cloud (definition: a landmark point is the description of a point in the 3D coordinate system, and a pixel point is the description of a point in the 2D image coordinate system). At a certain time k, the position and posture of the downward camera are and , the camera projection model is , the projection model is described by the camera distortion model, focal length, resolution, eccentricity and other parameters, which can generally be obtained through calibration. For this landmark point, the process of projecting it to the image coordinate system can be expressed as .in, for 2D coordinates in image coordinate system.

[0085] Furthermore, the processing module of the GNSS integrated navigation system is used for:

[0086] Acquire an initial top view of the target area using the three-dimensional point cloud data of the target area, and perform bilinear interpolation on the initial top view to acquire the top view;

[0087] or,

[0088] The image matching of the pixels of the landmark point in the overhead view with the pixels in the image data comprises:

[0089] Perform bilinear interpolation on the pixels in the area where the landmark points in the top view are located;

[0090] Obtain image matching between the locations of landmark points in the interpolated top view and those in the image data.

[0091] Since the 3D point cloud is generated by long-distance shooting, the resulting pixel density is usually lower than the actual resolution of the downward-looking camera when projected at a close distance. In this embodiment, the above 2D pixel clusters are further bilinearly interpolated to meet the pixel density of the downward-looking camera.

[0092] The processing module of the GNSS integrated navigation system is used for:

[0093] Using the expression formula, a cluster of top-view coordinates of landmark points are obtained;

[0094] A cluster of top view coordinates and a cluster of image data are subjected to the image matching.

[0095] Specifically, the processing module of the GNSS integrated navigation system is used for:

[0096] Using the image matching to construct the estimation problem of the position and posture data deviation of the downward-looking lens;

[0097] Minimizing the estimation problem to obtain a bias correction amount;

[0098] The deviation correction amount is used to obtain the real-time coordinates of the landmark points in the image data.

[0099] The estimation problem is:

[0100] ,

[0101] in, is the estimated value of the position and attitude deviation of the downward-looking lens, is the position and attitude deviation of the downward-looking lens, is a cluster of top-view coordinates of landmark points, is a cluster of image data of landmark points, is the luminosity value of the pixel position of the landmark point in the image data, is the inverse transformation of the camera projection model.

[0102] This embodiment uses a direct method for pixel-level image matching, that is, given a series of projected and interpolated point cloud 2D coordinates , and a cluster of image pixel values ​​taken by the downward-looking camera , the matching of two clusters of 2D point clouds can be achieved by minimizing the breadth error, that is, .

[0103] in, is the estimated value of position and attitude deviation, is the luminosity value of the pixel position, is the inverse transformation of the camera projection model.

[0104] By solving the above minimization problem, the deviation between the expected camera pose and the actual camera pose can be obtained. Based on this, the point to be staked can be reprojected into the image, and its projection position becomes .

[0105] use Get the real-time coordinates of landmark points in image data, is the position of the landmark point in the image data, and the visual lofting method includes: performing AR real-scene display of the real-time coordinates.

[0106] By displaying the above-mentioned projected 2D coordinates in AR real scene, a more accurate layout position can be expressed.

[0107] The processing module of the GNSS integrated navigation system is used for:

[0108] When moving away from the target area, a plurality of frames of dynamic images of the target area are captured by a front-view camera to obtain position and attitude data of the GNSS integrated navigation system during the process of moving to the target area;

[0109] When approaching the target area, the downward-looking lens is used to obtain image data of the target area, and the position and attitude data of the GNSS integrated navigation system during the process of moving to the target area;

[0110] Acquire three-dimensional point cloud data of the target area using the plurality of frames of dynamic images;

[0111] See also Figure 3 , using the above GNSS integrated navigation system, this embodiment also provides a visual stakeout method, including:

[0112] Step 100, obtaining three-dimensional point cloud data of a target area;

[0113] Step 101: Acquire a top view of the target area using the three-dimensional point cloud data of the target area;

[0114] Step 102: Acquire position and attitude data of the GNSS integrated navigation system during its movement to the target area and image data of the target area;

[0115] Step 103: for a road sign in the target area, image matching is performed between pixels of the road sign in the top view and pixels in the image data;

[0116] Step 104: using the image matching to obtain a deviation correction amount between the actual position and posture of the downward-looking lens and the position and posture estimated by the GNSS integrated navigation system;

[0117] Step 105: Obtain the coordinates of the landmark points in the image data using the correction amount of the position and attitude data of the GNSS integrated navigation system.

[0118] The GNSS integrated navigation system includes a downward-looking lens for shooting downward, the image data is image data shot by the downward-looking lens, and step 103 includes:

[0119] Step 1031, obtaining three-dimensional point cloud data of landmark points;

[0120] Step 1032: for a target moment, obtain the position and posture data of the downward-looking lens and the camera projection model;

[0121] Step 1033: using the position and posture data of the downward-looking lens and the camera projection model, establish a formula for projecting the three-dimensional point cloud data onto the top view;

[0122] Step 1034: Use the expression formula to perform image matching between the pixels of the landmark point in the overhead view and the pixels in the image data.

[0123] Wherein, the expression formula is:

[0124] ,

[0125] in, is the position data of the downward-looking lens, is the posture data of the downward-looking lens, is the three-dimensional point cloud data of landmark points. is the camera projection model, is the position of the landmark point in the top view.

[0126] Specifically, step 101 includes:

[0127] Acquire an initial top view of the target area using the three-dimensional point cloud data of the target area, and perform bilinear interpolation on the initial top view to acquire the top view;

[0128] In other implementations, interpolation may not be performed on the entire image, but only on key parts, that is:

[0129] The image matching of the pixels of the landmark point in the overhead view with the pixels in the image data comprises:

[0130] Perform bilinear interpolation on the pixels in the area where the landmark points in the top view are located;

[0131] Obtain image matching between the locations of landmark points in the interpolated top view and those in the image data.

[0132] The step 1034 includes:

[0133] Using the expression formula, a cluster of top-view coordinates of landmark points are obtained;

[0134] A cluster of top view coordinates and a cluster of image data are subjected to the image matching.

[0135] Specifically, the step 104 includes:

[0136] Using the image matching to construct the estimation problem of the position and posture data deviation of the downward-looking lens;

[0137] Step 105 includes:

[0138] Minimizing the estimation problem to obtain a bias correction amount;

[0139] The deviation correction amount is used to obtain the real-time coordinates of the landmark points in the image data.

[0140] The estimation problem is:

[0141] ,

[0142] in, is the estimated value of the position and attitude deviation of the downward-looking lens, is a cluster of top-view coordinates of landmark points, is a cluster of image data of landmark points, is the luminosity value of the pixel position of the landmark point in the image data, is the inverse transformation of the camera projection model.

[0143] In step 105, using Get the real-time coordinates of landmark points in image data, is the position of the landmark point in the image data, and the visual lofting method further includes:

[0144] Step 106: Display the real-time coordinates in an AR real scene.

[0145] The visual lofting method comprises:

[0146] When moving away from the target area, a plurality of frames of dynamic images of the target area are captured by using a front-view camera to obtain position and attitude data of the GNSS integrated navigation system during the process of moving to the target area, which can be achieved in step 100 .

[0147] When approaching the target area, the downward-looking lens is used to obtain image data of the target area, and the position and attitude data of the GNSS integrated navigation system during the process of moving to the target area;

[0148] Acquiring the three-dimensional point cloud data of the target area using the plurality of frames of dynamic images can be implemented in step 100 .

[0149] This embodiment can improve the positioning accuracy near the obstruction, and improve the positioning accuracy of the device from the decimeter level to the centimeter level.

[0150] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A visual lofting method based on image matching, used in a GNSS integrated navigation system, characterized in that: The GNSS integrated navigation system includes a downward-looking lens for shooting downward, and the visual lofting method includes: Acquire three-dimensional point cloud data of a target area; Acquire a top view of the target area using the three-dimensional point cloud data of the target area; Acquire position and attitude data of the GNSS integrated navigation system near the target area and image data of the target area, wherein the image data is image data taken by a downward-looking lens; For a landmark point in the target area, image matching is performed between pixels of the landmark point in the overhead view and pixels in the image data; Using the image matching, the deviation correction amount between the real position and posture of the downward-looking lens and the position and posture estimated by the GNSS integrated navigation system is obtained; The coordinates of the landmark points in the image data are obtained by using the correction amount of the position and attitude data of the GNSS integrated navigation system.

2. The visual lofting method based on image matching according to claim 1, characterized in that: The step of performing image matching on the pixels of the landmark point in the overhead view and the pixels in the image data comprises: Obtain three-dimensional point cloud data of landmark points; For a target moment, obtain the position and posture data of the downward-looking lens and the camera projection model; The position and posture data of the downward-looking lens and the camera projection model are used to establish a formula for projecting the 3D point cloud data onto the overhead view. The expression formula is used to perform image matching between the pixels of the landmark point in the overhead view and the pixels in the image data. The pixels of the landmark point in the overhead view and the pixels in the image data are matched with each other.

3. The visual lofting method based on image matching as claimed in claim 2, characterized in that: The expression formula is: ， in, is the position data of the downward-looking lens, is the posture data of the downward-looking lens, is the 3D point cloud data of landmark points. is the camera projection model, is the location of the landmark point in the top view.

4. The visual lofting method based on image matching as claimed in claim 3, characterized in that: The step of obtaining a top view of the target area by using the three-dimensional point cloud data of the target area includes: Acquire an initial top view of the target area using the three-dimensional point cloud data of the target area, and perform bilinear interpolation on the initial top view to acquire the top view; or, The image matching of the pixels of the landmark point in the overhead view with the pixels in the image data comprises: Perform bilinear interpolation on the pixels in the area where the landmark points in the top view are located; Obtain image matching between the locations of landmark points in the interpolated top view and those in the image data.

5. The visual lofting method based on image matching as claimed in claim 3, characterized in that: The step of using the expression formula to perform image matching between pixels of the landmark point in the overhead view and pixels in the image data includes: Using the expression formula, a cluster of top-view coordinates of landmark points are obtained; A cluster of top view coordinates and a cluster of image data are subjected to the image matching.

6. The visual lofting method based on image matching as claimed in claim 5, characterized in that: The visual lofting method comprises: Using the image matching to construct the estimation problem of the position and posture data deviation of the downward-looking lens; Minimizing the estimation problem to obtain the deviation correction amount; The real-time coordinates of the landmark points in the image data are obtained using the deviation correction data.

7. The visual lofting method based on image matching according to claim 6, characterized in that: The estimation problem is: ， in, is the estimated value of the position and attitude deviation of the downward-looking lens, is a cluster of top-view coordinates of landmark points, is a cluster of image data of landmark points, is the luminosity value of the pixel position of the landmark point in the image data, is the inverse transformation of the camera projection model, is the position and attitude deviation of the downward-looking lens, Solve the symbol for the minimization of the estimation problem.

8. The visual lofting method based on image matching according to claim 6, characterized in that: use Get the real-time coordinates of landmark points in image data, is the location of the landmark point in the image data, is the camera projection model, is the posture data of the downward-looking lens, is the position data of the downward-looking lens, is the 3D point cloud data of landmark points. is the estimated value of the position and attitude deviation of the downward-looking lens, The position and posture deviation of the downward-looking lens, the visual lofting method includes: displaying the real-time coordinates in an AR real scene.

9. The visual lofting method based on image matching as claimed in claim 2, characterized in that: The GNSS integrated navigation system includes a forward-looking lens, and the visual lofting method includes: When moving away from the target area, a plurality of frames of dynamic images of the target area are captured by a front-view camera to obtain position and attitude data of the GNSS integrated navigation system during the process of moving to the target area; When approaching the target area, the downward-looking lens is used to obtain image data of the target area, and the position and attitude data of the GNSS integrated navigation system during the process of moving to the target area; The three-dimensional point cloud data of the target area is acquired by using the several frames of dynamic images.

10. A GNSS integrated navigation system, characterized in that: The GNSS integrated navigation system is used to implement the visual layout method based on image matching as described in any one of claims 1 to 9.

Images