Key frame extraction method for vision measurement and integrated navigation system

Abstract

The invention discloses a key frame extraction method for vision measurement and an integrated navigation system, and the method comprises the steps: carrying out the vision measurement of a target point through employing the integrated navigation system, and obtaining a plurality of image data related to the target point; and obtaining a key frame image in the image data according to the motion state of the integrated navigation system, wherein the key frame image is used for aerial triangulation calculation of the vision measurement to obtain the position coordinates of the target point. According to the method, the photos which conform to a certain overlapping degree and are low in repeatability can be extracted to serve as the key frames, so that the precision and efficiency of subsequent photo point selection measurement are improved.

Description

Technical Field

[0001] The present invention relates to the field of GNSS integrated navigation systems, and in particular to a key frame extraction method for visual measurement and an integrated navigation system. Background Art

[0002] GNSS-based engineering surveying technology has developed over decades, from centering measurement to the tilt measurement that is widely used today. In recent years, the release of GNSS integrated navigation systems has provided a new direction for engineering surveying technology.

[0003] Currently, RTK combined with GNSS positioning, IMU attitude and visual measurement of camera photos has been widely used. Through the precise positioning position after RTK fixation and the attitude information output by the built-in IMU, combined with the parameters of the installed camera, the position and attitude of the camera taking photos at any time can be determined. Selecting the same target point on multiple photos and calculating through aerial triangulation can obtain the exact position of the target point. It is already a very mature application and is implemented on many RTK devices.

[0004] Since the camera can take ten or twenty original pictures per second, if the target point of interest is taken continuously for about ten seconds, hundreds of original photos can be obtained. In the process of photo point selection and measurement, it is generally only necessary to select points in a few photos for measurement, and the coordinates of the target point can be obtained based on the aerial triangulation relationship. Among the many current implementation schemes, the scheme of selecting photos is relatively simple, basically selecting one or two photos per second and presenting them to the user for the user to select points on the photos for measurement. This scheme has obvious disadvantages. When the RTK moves slowly and the change in viewing angle per second is not obvious, the repetition of the photos directly selected according to time may be too large. Selecting points on photos with high repetition does not help the visual measurement accuracy at all, and long-term shooting may result in too many photos with high repetition, which brings great inconvenience to the point selection and measurement operation. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the defects of high repeatability and unclear viewing angle change of photos used for aerial triangulation in the prior art, and to provide a key frame extraction method and an integrated navigation system for visual measurement that can extract photos that meet a certain overlap and have low repeatability as key frames, thereby improving the accuracy and efficiency of subsequent photo point selection measurement.

[0006] The present invention solves the above technical problems through the following technical solutions:

[0007] A key frame extraction method for visual measurement, characterized in that the key frame extraction method comprises:

[0008] Use the integrated navigation system to perform visual measurement on the target point and obtain a number of image data related to the target point;

[0009] The key frame images in the image data are acquired according to the motion state of the integrated navigation system, and the key frame images are used for the aerial triangulation calculation of the visual measurement to acquire the position coordinates of the target point.

[0010] Preferably, the key frame extraction method comprises:

[0011] In the visual measurement, the integrated navigation system acquires positioning data and attitude data;

[0012] Acquire the position and posture information of the camera using the positioning data, the posture data and the parameters of the camera in the integrated navigation system;

[0013] The position coordinates of the target point are obtained by performing aerial triangulation calculation using the position information and the key frame images.

[0014] Preferably, the step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes:

[0015] The moving speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system;

[0016] The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

[0017] Preferably, the moving speed is a positioning change per unit time of a GNSS module in the integrated navigation system.

[0018] Preferably, the step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes:

[0019] The rotation speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system;

[0020] The key frame images in the image data are acquired according to the rotation speed, wherein the faster the rotation speed is, the smaller the time interval between adjacent key frame images is.

[0021] Preferably, the step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes:

[0022] The GNSS module of the integrated navigation system is used to obtain the camera positioning data, the IMU module of the integrated navigation system is used to obtain the camera attitude data, and the camera parameters of the integrated navigation system are used to obtain the shooting direction;

[0023] Using the positioning data and the attitude data, a velocity component of the sum of the camera position movement velocity and the camera swing velocity on a target plane is obtained, wherein the target plane is perpendicular to the shooting direction;

[0024] Obtaining the moving speed of the camera using the speed component and the camera rotation speed;

[0025] The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

[0026] Preferably, the key frame extraction method comprises:

[0027] Acquire an initial key frame image in the image data according to the motion state of the integrated navigation system;

[0028] Identify feature points in the initial key frame image;

[0029] Determine whether the feature point overlap rate of adjacent initial key frame images meets a preset value, and if not, replace at least one of the adjacent initial key frame images with an optimized frame image, wherein the optimized frame image is obtained by searching before and after the initial key frame image that does not meet the preset value;

[0030] The initial key frame image and the optimized frame image that meet the preset values ​​are used as the final key frame image.

[0031] Preferably, the key frame extraction method comprises:

[0032] For the first frame image, determine whether another adjacent initial key frame image of the first frame image is the final key frame image. If so, search for the target frame image with the highest overlap rate of feature points with the second frame image between the adjacent initial key frame images other than the second frame image and the first frame image as the final key frame image. Otherwise, take the other adjacent initial key frame image as the first frame image and then execute again the step of determining whether the other adjacent initial key frame image of the first frame image is the final key frame image, wherein the first frame image and the second frame image are adjacent initial key frame images that do not meet the preset value.

[0033] The present invention also provides an integrated navigation system, which is characterized in that the integrated navigation system is used to implement the key frame extraction method for visual measurement as described above.

[0034] Preferably, the integrated navigation system is a GNSS receiver.

[0035] On the basis of being in accordance with the common sense in the art, the above-mentioned preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present invention.

[0036] The positive and progressive effects of the present invention are:

[0037] During the visual measurement shooting process, the speed of RTK movement is not fixed, so two photos taken one second apart may be almost identical with a high degree of repetition, or may have a very low degree of overlap and obvious changes in the objects in the photos. In the visual measurement selection photo measurement, a certain number of photos need to be selected to improve the efficiency of subsequent point selection measurement operations on the photos. On the one hand, this requires a certain degree of overlap between adjacent photos to satisfy the aerial triangulation relationship for calculation; on the other hand, it requires that when selecting photos containing target points of interest for measurement, the perspectives of the target points of interest in the photos should be as different as possible, so that the accuracy of obtaining the target points of interest can be high enough.

[0038] The present invention extracts key frames in visual measurement. The key frames are not simply extracted according to time intervals. Instead, a certain method is used to extract photos that meet a certain overlap and have low repeatability as key frames when the RTK movement speed is fast or slow, so as to improve the accuracy and efficiency of subsequent photo point selection measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of the key frame extraction method of embodiment 1 of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0041] Example 1

[0042] This embodiment provides a GNSS integrated navigation system, which includes a GNSS module, an IMU module, a processing module and a camera module. The GNSS module, the IMU module and the camera module can be integrated in a GNSS receiver, that is, the GNSS receiver is the GNSS integrated navigation system.

[0043] Integrated navigation systems are used for:

[0044] Perform visual measurement on the target point, and the camera module is used to obtain a number of image data related to the target point;

[0045] The processing module is used to obtain key frame images in the image data according to the motion state of the integrated navigation system, and the key frame images are used for the aerial triangulation calculation of the visual measurement to obtain the position coordinates of the target point.

[0046] In the visual measurement, the integrated navigation system is used to obtain positioning data and attitude data, specifically, the GNSS module is used to obtain positioning data, and the IMU module is used to obtain attitude data;

[0047] The processing module is used to obtain the position and posture information of the camera by using the positioning data, the posture data and the parameters of the camera in the integrated navigation system;

[0048] The processing module is also used to use the position information and the key frame image to perform aerial triangulation calculation to obtain the position coordinates of the target point.

[0049] The processing module is used to:

[0050] The moving speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system;

[0051] The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

[0052] The moving speed is the positioning change per unit time of the GNSS module in the integrated navigation system.

[0053] The processing module is used to:

[0054] The rotation speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system;

[0055] The key frame images in the image data are acquired according to the rotation speed, wherein the faster the rotation speed is, the smaller the time interval between adjacent key frame images is.

[0056] Specifically, the processing module is used to:

[0057] The GNSS module of the integrated navigation system is used to obtain the camera positioning data, the IMU module of the integrated navigation system is used to obtain the camera attitude data, and the camera parameters of the integrated navigation system are used to obtain the shooting direction;

[0058] Using the positioning data and the attitude data, a velocity component of the sum of the camera position movement velocity and the camera swing velocity on a target plane is obtained, wherein the target plane is perpendicular to the shooting direction;

[0059] The speed of the camera can be regarded as the moving speed of the bottom of the centering pole plus the swinging speed of the camera on the centering pole. The moving direction should be considered here. For example, if the centering pole moves to the left and the camera swings to the right, the speed values ​​are subtracted. If they move in the same phase, the speed values ​​are added.

[0060] The velocity component is small when moving forward and backward relative to the target point, and the difference in viewing angle is considered small.

[0061] Obtaining the moving speed of the camera using the speed component and the camera rotation speed;

[0062] The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

[0063] Specifically, the processing module is used to:

[0064] Acquire an initial key frame image in the image data according to the motion state of the integrated navigation system;

[0065] Identify feature points in the initial key frame image;

[0066] Determine whether the feature point overlap rate of adjacent initial key frame images meets a preset value, and if not, replace at least one of the adjacent initial key frame images with an optimized frame image, wherein the optimized frame image is obtained by searching before and after the initial key frame image that does not meet the preset value;

[0067] The initial key frame image and the optimized frame image that meet the preset values ​​are used as the final key frame image.

[0068] Furthermore, the processing module is used to:

[0069] For the first frame image, determine whether another adjacent initial key frame image of the first frame image is the final key frame image. If so, search for the target frame image with the highest overlap rate of feature points with the second frame image between the adjacent initial key frame images other than the second frame image and the first frame image as the final key frame image. Otherwise, take the other adjacent initial key frame image as the first frame image and then execute again the step of determining whether the other adjacent initial key frame image of the first frame image is the final key frame image, wherein the first frame image and the second frame image are adjacent initial key frame images that do not meet the preset value.

[0070] The algorithm for extracting key frames from visual measurements is mainly based on two aspects: camera motion judgment and photo element judgment. The camera motion judgment is the basis for calculating the photo triangulation principle. The camera motion can ensure that new information continues to appear in the photo, thereby ensuring that the extraction of key frames has measurement significance. The judgment of photo element can further verify the new information. The judgment of photo element cannot be independent of the camera motion judgment. Otherwise, when shooting features with repeated textures, abnormal problems are likely to occur in the key frame extraction algorithm.

[0071] One of the indicators for judging the camera motion is the camera's translational motion, which is mainly based on the change in GNSS positioning. When the camera's shooting posture and angle remain unchanged, the movement of the camera center causes the movement of the photo center, so that the key frame can be extracted based on the change in movement. Generally speaking, when the movement is fast, the time interval between adjacent key frames is short to ensure that adjacent key frames meet a certain degree of overlap. When the target point of interest appears in the overlapping area of ​​adjacent key frames, the key frames can constitute the aerial triangulation to calculate the coordinates of the target point of interest. When the movement is slow, the time interval between adjacent key frames is long to ensure that the repetition of adjacent key frames is small. In this way, selecting points of interest for measurement within a certain number of key frames can ensure that the parallax of different photos is large, and the observation independence of each adjacent key frame is large. A high-precision result of the target point of interest can be determined through a certain number of key frame photos.

[0072] The camera motion judgment also includes the camera rotation, which is mainly based on the change of the heading angle indicator of the IMU module. When the RTK position change is not obvious, the rotation of the device allows the camera installed on the side to shoot at different angles, and the key frames are extracted according to the change in rotation speed. Similarly, when the heading angle changes quickly, the time interval between adjacent key frames is short to ensure that the adjacent key frames meet a certain overlap. When the heading angle changes slowly, the time interval between adjacent key frames is long to ensure that the repetition of adjacent key frames is small.

[0073] The judgment of photo shooting elements needs to be based on the extraction of image feature points, and the changes in the information of the photographed object are confirmed according to the changes in the feature points. There are many ways to extract image feature points. According to the grayscale / color value of the image pixels, the positions where the grayscale / color value of adjacent pixels changes significantly are recorded as feature points. Generally, these feature points are located on the obvious texture pattern of the photographed object in the photo. When new information continues to appear in the photo, generally these extracted image feature points will undergo overall displacement and rotation on the image plane, and the number of extracted image feature points will also change continuously, thereby ensuring that the primitives in the key frame meet the requirements of key frame extraction.

[0074] Through the key frame extraction of visual measurement, when a target point of interest is photographed, regardless of the moving speed, for example, one surveyor shoots for 10 seconds and another surveyor shoots for 20 seconds, the number of key frames extracted is similar, and the key frames can meet the calculation requirements of the visual measurement triangulation principle and form sufficient parallax angles to calculate the position of the target point of interest with high precision.

[0075] See also Figure 1 , using the above-mentioned combined navigation system, this embodiment further provides a key frame extraction method for visual measurement, characterized in that the key frame extraction method comprises:

[0076] Step 100: Use the integrated navigation system to perform visual measurement on the target point to obtain a number of image data related to the target point;

[0077] Step 101: acquiring key frame images in image data according to the motion state of the integrated navigation system, wherein the key frame images are used for aerial triangulation calculation of the visual measurement to acquire the position coordinates of the target point.

[0078] During the execution of step 100, the following can also be executed simultaneously:

[0079] In the visual measurement, the integrated navigation system acquires positioning data and attitude data;

[0080] Acquire the position and posture information of the camera using the positioning data, the posture data and the parameters of the camera in the integrated navigation system;

[0081] Step 102: Perform aerial triangulation calculation using the position information and the key frame image to obtain the position coordinates of the target point.

[0082] Step 101 includes:

[0083] Step 1011: Acquire the moving speed of the camera in the integrated navigation system by using the position information of the integrated navigation system;

[0084] Step 1012: using the position information of the integrated navigation system to obtain the rotation speed of the camera in the integrated navigation system;

[0085] Step 1013: Acquire key frame images in the image data according to the moving speed and the rotation speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is, and the faster the rotation speed is, the smaller the time interval between adjacent key frame images is.

[0086] The moving speed is the positioning change per unit time of the GNSS module in the integrated navigation system.

[0087] In other implementations, step 101 includes:

[0088] The GNSS module of the integrated navigation system is used to obtain the camera positioning data, the IMU module of the integrated navigation system is used to obtain the camera attitude data, and the camera parameters of the integrated navigation system are used to obtain the shooting direction;

[0089] Using the positioning data and the attitude data, a velocity component of the sum of the camera position movement velocity and the camera swing velocity on a target plane is obtained, wherein the target plane is perpendicular to the shooting direction;

[0090] Obtaining the moving speed of the camera using the speed component and the camera rotation speed;

[0091] The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

[0092] Step 101 also includes:

[0093] Acquire an initial key frame image in the image data according to the motion state of the integrated navigation system;

[0094] Identify feature points in the initial key frame image;

[0095] Determine whether the feature point overlap rate of adjacent initial key frame images meets a preset value, and if not, replace at least one of the adjacent initial key frame images with an optimized frame image, wherein the optimized frame image is obtained by searching before and after the initial key frame image that does not meet the preset value;

[0096] The initial key frame image and the optimized frame image that meet the preset values ​​are used as the final key frame image.

[0097] Among them, for the first frame image, determine whether another adjacent initial key frame image of the first frame image is the final key frame image. If so, search for the target frame image with the highest overlap rate of feature points with the second frame image between the adjacent initial key frame images other than the second frame image and the first frame image as the final key frame image. If not, take the other adjacent initial key frame image as the first frame image and then execute again the step of determining whether the other adjacent initial key frame image of the first frame image is the final key frame image, wherein the first frame image and the second frame image are adjacent initial key frame images that do not meet the preset value.

[0098] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that these are only examples, and the protection scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but these changes and modifications all fall within the protection scope of the present invention.

Claims

1. A key frame extraction method for visual measurement, characterized in that: The key frame extraction method comprises: Use the integrated navigation system to perform visual measurement on the target point and obtain a number of image data related to the target point; The key frame images in the image data are acquired according to the motion state of the integrated navigation system, and the key frame images are used for the aerial triangulation calculation of the visual measurement to acquire the position coordinates of the target point.

2. The key frame extraction method for visual measurement according to claim 1, characterized in that: The key frame extraction method comprises: In the visual measurement, the integrated navigation system acquires positioning data and attitude data; Acquire the position and posture information of the camera using the positioning data, the posture data and the parameters of the camera in the integrated navigation system; The position coordinates of the target point are obtained by performing aerial triangulation calculation using the position information and the key frame images.

3. The key frame extraction method for visual measurement according to claim 1, characterized in that: The step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes: The moving speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system; The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

4. The key frame extraction method for visual measurement according to claim 3, characterized in that: The moving speed is the positioning change per unit time of the GNSS module in the integrated navigation system.

5. The key frame extraction method for visual measurement according to claim 1, characterized in that: The step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes: The rotation speed of the camera in the integrated navigation system is obtained by using the position information of the integrated navigation system; The key frame images in the image data are acquired according to the rotation speed, wherein the faster the rotation speed is, the smaller the time interval between adjacent key frame images is.

6. The key frame extraction method for visual measurement according to claim 1, characterized in that: The step of acquiring key frame images in the image data according to the motion state of the integrated navigation system includes: The GNSS module of the integrated navigation system is used to obtain the camera positioning data, the IMU module of the integrated navigation system is used to obtain the camera attitude data, and the camera parameters of the integrated navigation system are used to obtain the shooting direction; Using the positioning data and the attitude data, a velocity component of the sum of the camera position movement velocity and the camera swing velocity on a target plane is obtained, wherein the target plane is perpendicular to the shooting direction; Obtaining the moving speed of the camera using the speed component and the camera rotation speed; The key frame images in the image data are acquired according to the moving speed, wherein the faster the moving speed is, the smaller the time interval between adjacent key frame images is.

7. The key frame extraction method for visual measurement according to any one of claims 1 to 6, characterized in that: The key frame extraction method comprises: Acquire an initial key frame image in the image data according to the motion state of the integrated navigation system; Identify feature points in the initial key frame image; Determine whether the feature point overlap rate of adjacent initial key frame images meets a preset value, and if not, replace at least one of the adjacent initial key frame images with an optimized frame image, wherein the optimized frame image is obtained by searching before and after the initial key frame image that does not meet the preset value; The initial key frame image and the optimized frame image that meet the preset values ​​are used as the final key frame image.

8. The key frame extraction method for visual measurement according to claim 7, characterized in that: The key frame extraction method comprises: For the first frame image, determine whether another adjacent initial key frame image of the first frame image is the final key frame image. If so, search for the target frame image with the highest overlap rate of feature points with the second frame image between the adjacent initial key frame images other than the second frame image and the first frame image as the final key frame image. Otherwise, take the other adjacent initial key frame image as the first frame image and then execute again the step of determining whether the other adjacent initial key frame image of the first frame image is the final key frame image, wherein the first frame image and the second frame image are adjacent initial key frame images that do not meet the preset value.

9. An integrated navigation system, characterized in that: The integrated navigation system is used to implement the key frame extraction method for visual measurement as described in any one of claims 1 to 8.

10. The integrated navigation system according to claim 9, characterized in that: The integrated navigation system is a GNSS receiver.

Images