Retrieval truth value acquisition method, image retrieval method and system

Abstract

The present invention provides a method for obtaining a retrieval truth value, an image retrieval method and a system, wherein the method for obtaining a retrieval truth value comprises: obtaining a visual map and its map keyframes, and a first three-dimensional coordinate set; collecting continuous image frames, aligning the continuous image frames to the visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames; generating a corresponding first bounding box set and a second bounding box for the map keyframes and the first three-dimensional coordinate set and the second three-dimensional coordinate set corresponding to the continuous image frames; and selecting the map keyframe as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame according to a first ratio of overlapping bounding boxes of the second bounding box set relative to the first bounding box set to all second bounding boxes. The technical solution of the present invention can improve the robustness and accuracy of the image retrieval model.

Description

Technical Field

[0001] The present invention relates to the field of computer computing, and in particular to a retrieval truth value acquisition method, an image retrieval method and a system. Background Art

[0002] Image retrieval based on visual positioning requires a certain method to find the image most similar to the image to be retrieved from the existing image library. In the prior art, these methods are usually local feature re-aggregation methods, or global descriptor methods based on deep learning. Local feature re-aggregation methods specifically include the BOW (Bagof words) method based on traditional manual feature SIFT aggregation, the DELG algorithm based on deep learning to extract local features and aggregate them, etc. Global descriptor methods based on deep learning include NetVLAD, GeM and other methods.

[0003] Using the above method to achieve image retrieval, the feature aggregation ability based on deep learning is better than the manual feature aggregation ability. Regardless of the global descriptor used, the computer needs to be self-trained according to the built algorithm model to obtain the optimal parameters of the model. When the computer is self-training, it needs to provide corresponding data, construct the relationship between the data, and improve the robustness and accuracy of the model through supervised training. Whether in model training or evaluating the trained model, good data values ​​are required.

[0004] At present, the relationship between the data sets generally used to train the model is to roughly determine whether there is a visual relationship between the images through the GPS values ​​between the images themselves. This judgment method will cause defects in the existing image retrieval method based on visual positioning. There are often many erroneous relationships in the positional relationship between the images obtained by GPS. The numerical accuracy of GPS itself is not fixed, and it is difficult to distinguish whether there is a correlation between the retrieval image and the database image through a fixed threshold. This makes the correlation judgment between the retrieval image and the database image wrong. The wrong judgment of image correlation will affect the model parameters during the training process, causing these parameters to become continuously wrong. In the selection of the optimal model, a not-so-good model may also be selected, and the robustness and accuracy of the image retrieval results are very low. Summary of the invention

[0005] The technical problem solved by the technical solution of the present invention is: how to construct an evaluation technology for numerical accuracy in image retrieval based on visual positioning in the prior art to improve the robustness and accuracy of the image retrieval model.

[0006] In order to solve the above technical problems, the technical solution of the present invention provides a method for obtaining a retrieval truth value based on continuous image frames, comprising:

[0007] Acquire a visual map, wherein the visual map includes a map key frame and a first three-dimensional coordinate set corresponding to the map key frame;

[0008] Acquire continuous image frames, and align the continuous image frames to a visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames;

[0009] Generate a first bounding box set for the first three-dimensional coordinate set corresponding to the map key frame, wherein the first bounding box set includes a first bounding box for each first three-dimensional coordinate;

[0010] generating a second bounding box set for a second three-dimensional coordinate set corresponding to the continuous image frames, the second bounding box set including a second bounding box for each second three-dimensional coordinate;

[0011] If the second bounding box set has overlapping bounding boxes relative to the first bounding box set, the corresponding map key frame is selected as the retrieval truth value of the continuous image frame based on a first ratio of the overlapping bounding boxes to all second bounding boxes to obtain a co-viewing image set of the continuous image frames.

[0012] Optionally, selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes:

[0013] If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates;

[0014] If the second ratio and the third ratio are both greater than the second threshold, and the orientation angle between the continuous image frame and the map key frame is less than 90°, the map key frame is selected as the retrieval truth value of the continuous image frame to obtain the co-viewed image set of the continuous image frame.

[0015] Optionally, selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes:

[0016] If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates;

[0017] If the second ratio and the third ratio are both greater than a second threshold, and the orientation angle between the continuous image frame and the map key frame is less than 90°, the map key frame is selected as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame;

[0018] If at least one of the second ratio and the third ratio is less than or equal to the second threshold, then when the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold, the map key frame is selected as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame.

[0019] Optionally, selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes:

[0020] If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates;

[0021] If at least one of the second ratio and the third ratio is less than or equal to the second threshold, then when the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold, the map key frame is selected as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame.

[0022] Optionally, selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes:

[0023] If the first ratio is greater than a first threshold, the map key frame is selected as a retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frames.

[0024] Optionally, the first three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the map key frame, and the second three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the continuous image frame.

[0025] Optionally, the second three-dimensional coordinate set corresponding to the continuous image frames is obtained by the following steps:

[0026] Acquire a first relative position and posture of the continuous image frames, wherein the first relative position and posture includes a first rotation angle and a first relative displacement of the continuous image frames relative to a reference image;

[0027] Collecting a second relative position of the continuous image frames in the coordinate system of the visual map, wherein the second relative position includes a second rotation angle and a second relative displacement obtained by positioning the continuous image frames in the visual map;

[0028] Calibrate a third relative posture of the continuous image frames in the visual map by using the first relative posture and the second relative posture of the continuous image frames, wherein the third relative posture includes a third rotation angle and three relative displacements obtained by calibrating the continuous image frames in the visual map;

[0029] Based on the third relative positions of all the consecutive image frames, a retrieval map is obtained by triangulation reconstruction, wherein the retrieval map contains a second three-dimensional coordinate set corresponding to the consecutive image frames.

[0030] Optionally, acquiring the first relative posture of the continuous image frames includes:

[0031] The first relative position and posture of the continuous image frames are obtained by using a VIO algorithm or a SLAM algorithm.

[0032] Optionally, the collecting of the second relative posture of the continuous image frames in the coordinate system where the visual map is located includes:

[0033] Performing feature matching between the continuous image frames and the visual map to obtain a matching three-dimensional coordinate set in a first three-dimensional coordinate set;

[0034] Based on the matching three-dimensional coordinate set, a second relative posture corresponding to the continuous image frames is obtained through a PNP algorithm.

[0035] Optionally, calibrating a third relative pose of the continuous image frames in the visual map by using the first relative pose and the second relative pose of the continuous image frames includes:

[0036] On the basis of the second relative pose of the continuous image frames, according to the first relative pose of the continuous image frames, all the continuous image frames are re-incorporated into the visual map for pose calibration using the RANSAC algorithm to obtain the third relative pose.

[0037] Optionally, the first bounding box and the second bounding box are AABB bounding boxes, bounding spheres, directional bounding boxes, or fixed-directional convex hulls.

[0038] Optionally, the method for obtaining a retrieval truth value based on continuous image frames further includes:

[0039] A collision detection is performed on the first bounding box and the second bounding box based on the first bounding box set and the second bounding box set to count a first ratio of the overlapping bounding boxes to all second bounding boxes.

[0040] In order to solve the above technical problems, the technical solution of the present invention further provides an image retrieval method based on continuous image frames, comprising:

[0041] Acquiring continuous image frames to be retrieved;

[0042] Based on the above-mentioned method for obtaining a retrieval truth value based on continuous image frames, a co-viewing image set corresponding to the continuous image frames is obtained;

[0043] The continuous image frames are queried by KNN based on global features to obtain the map key frame closest to the continuous image frames;

[0044] If the retrieved map key frame is the retrieval truth value of the co-visual image set corresponding to the continuous image frame, the retrieval is judged to be successful.

[0045] Optionally, performing feature matching between the continuous image frames and a map key frame of a visual map to retrieve a map key frame closest to the continuous image frames includes:

[0046] Extracting first global features and second global features of the map key frames and continuous image frames;

[0047] A matching first global feature and a map key frame corresponding to the first global feature are selected based on the second global feature of the continuous image frame, and the map key frame is the map key frame closest to the continuous image frame.

[0048] Optionally, the selecting of the matching first global feature and the map key frame corresponding to the first global feature based on the second global feature of the continuous image frames includes:

[0049] Based on the second global features of the continuous image frames, a matching first global feature and its corresponding map key frame are selected through a KNN query algorithm.

[0050] Optionally, the selecting of the matching first global feature and the map key frame corresponding to the first global feature based on the second global feature of the continuous image frames includes:

[0051] A first global feature and a corresponding map key frame are selected, the distance between the first global feature and the second global feature of the continuous image frames being less than or equal to a preset selection distance.

[0052] In order to solve the above technical problems, the technical solution of the present invention further provides a retrieval truth value acquisition system based on continuous image frames, comprising:

[0053] An acquisition unit, adapted to acquire a visual map, wherein the visual map comprises a map key frame and a first three-dimensional coordinate set corresponding to the map key frame;

[0054] A first acquisition unit, adapted to acquire continuous image frames, and align the continuous image frames to a visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames;

[0055] A first bounding box generating unit, adapted to generate a first bounding box set for a first three-dimensional coordinate set corresponding to the map key frame, wherein the first bounding box set includes a first bounding box for each first three-dimensional coordinate;

[0056] A second bounding box generating unit, adapted to generate a second bounding box set for a second three-dimensional coordinate set corresponding to the continuous image frames, the second bounding box set comprising a second bounding box for each second three-dimensional coordinate;

[0057] The selection unit is adapted to select the corresponding map key frame as the retrieval truth value of the continuous image frame based on a first ratio of the overlapping bounding boxes to all the second bounding boxes when the second bounding box set has overlapping bounding boxes relative to the first bounding box set to obtain the co-viewing image set of the continuous image frames.

[0058] Optionally, the selection unit includes:

[0059] a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold;

[0060] The first selection subunit is adapted to select the map key frame as the retrieval true value of the continuous image frame to obtain a co-viewing image set of the continuous image frame when the second ratio and the third ratio are both greater than a second threshold and the orientation angle between the continuous image frame and the map key frame is less than 90°.

[0061] Optionally, the selection unit includes:

[0062] a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold;

[0063] A first selection subunit is adapted to select the map key frame as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame when the second ratio and the third ratio are both greater than a second threshold and the orientation angle between the continuous image frame and the map key frame is less than 90°;

[0064] The second selection subunit is suitable for selecting the map key frame as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when at least one of the second ratio and the third ratio is less than or equal to the second threshold, the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold.

[0065] Optionally, the selection unit includes:

[0066] a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold;

[0067] The second selection subunit is suitable for selecting the map key frame as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when at least one of the second ratio and the third ratio is less than or equal to the second threshold, the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold.

[0068] Optionally, the selection unit includes:

[0069] The third selection subunit is adapted to select the map key frame as the retrieval truth value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when the first ratio is greater than a first threshold.

[0070] In order to solve the above technical problems, the technical solution of the present invention further provides an image retrieval system based on continuous image frames, comprising:

[0071] A second acquisition unit, adapted to acquire continuous image frames to be retrieved;

[0072] The retrieval truth value acquisition system as described above is suitable for obtaining a co-viewing image set corresponding to the collected continuous image frames to be retrieved;

[0073] A matching unit, adapted to obtain a map key frame closest to the continuous image frame through a KNN query based on the global features of the continuous image frame;

[0074] The judging unit is adapted to judge that the retrieval is successful when the retrieved map key frame is the retrieval truth value of the co-visual image set corresponding to the continuous image frames.

[0075] The beneficial effects of the technical solution of the present invention include at least:

[0076] The technical solution of the present invention can solve the problem of true value data in image retrieval based on visual positioning, and build a framework for constructing a new true value for visual positioning image retrieval. The data obtained by the framework can be used to train image retrieval methods for visual positioning, and at the same time, the accuracy of existing retrieval methods can be evaluated. Through automated operation, the problem of false true values ​​in the data can be solved, and manual intervention can be avoided. It can assist in outputting the retrieval true value and co-viewing relationship between the retrieval image and the database image, and improve the robustness and accuracy of the image retrieval model.

[0077] The technical solution of the present invention can obtain the position information of each frame image in the video sequence in the visual map for the generated visual map, and then obtain the common view relationship between the map frame image and the continuous image frame image according to the position relationship and overlap information between the three-dimensional point set corresponding to each map key frame in the visual map database and the three-dimensional point set of the continuous image frame image of the video sequence, and obtain the retrieval true value of each image frame. This technical solution can overcome the defects of the existing technology that the accuracy of GPS numerical values ​​is not fixed and it is difficult to retrieve the correlation between images, and can accurately judge the common view relationship of images, and improve the robustness and accuracy of the image retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0079] Figure 1 A data association diagram for constructing an image retrieval truth value and a corresponding co-viewing image set provided by the technical solution of the present invention;

[0080] Figure 2 A schematic flow chart of the steps of a method for obtaining a true retrieval value based on continuous image frames provided by the technical solution of the present invention;

[0081] Figure 3 A schematic flow chart of the steps of a method for obtaining a three-dimensional point cloud coordinate set of continuous image frames provided by the technical solution of the present invention;

[0082] Figure 4 A schematic flow chart of a variation of a method for obtaining a true retrieval value based on continuous image frames provided by the technical solution of the present invention;

[0083] Figure 5 A schematic flow chart of steps of another variation of the method for obtaining true retrieval values ​​based on continuous image frames provided by the technical solution of the present invention;

[0084] Figure 6 A schematic flow chart of steps of another variation of the method for obtaining a true retrieval value based on continuous image frames provided by the technical solution of the present invention;

[0085] Figure 7 A schematic flow chart of the steps of an image retrieval method based on continuous image frames provided by the technical solution of the present invention. DETAILED DESCRIPTION

[0086] In order to better illustrate the technical solution of the present invention, the present invention is further described below in conjunction with the accompanying drawings.

[0087] Image retrieval based on visual positioning requires searching for the image most similar to the image to be retrieved from the graphic library of the existing visual map. No matter what algorithm model is used when implementing image retrieval, it is necessary to provide training data for the computer, construct the relationship between the data, and improve the robustness and accuracy of the algorithm model.

[0088] Different from the prior art which uses image GPS values ​​to determine whether there is a visual relationship between images, the technical solution of the present invention provides a method for obtaining retrieval truth values ​​based on continuous image frames, which can collect continuous image frames through existing visual maps and their map key frames to construct new image retrieval truth values ​​and obtain a co-viewing image set corresponding to the continuous image frames. These data can be used to train image retrieval models for visual positioning, and at the same time, the accuracy of existing retrieval models can be evaluated. Through automated operation, the problem of false truth values ​​in the data can be solved, and manual intervention can be avoided.

[0089] To construct a new image retrieval truth value and the corresponding co-viewing image set, and obtain the co-viewing relationship, refer to Figure 1 The data frame diagram shown can be based on the following technical solution ideas:

[0090] Based on the generated visual map, a mobile device or other visible light device can collect one or more video sequences within the real space range corresponding to the visual map to form continuous image frames. The mobile device or other visible light device has the following sensors: gyroscope, accelerometer, magnetometer, GPS, etc., which can record the sensor information when collecting continuous images. Running VIO, SLAM or other methods based on the collected data can obtain the relative relationship between the continuous image frames. Match the continuous image frames with the existing visual map, and use the relative relationship between the continuous image frames to deeply couple all images and map key frames to obtain the position and posture information of all continuous image frames in the visual map, as well as the three-dimensional point cloud data corresponding to all continuous image frames. Using the visual map and the three-dimensional point cloud set of each frame of the continuous image, a series of methods can be used to obtain the robust common view relationship between the continuous image frames and the key frames of the visual map, and output the image retrieval truth value and the corresponding common view image set of the continuous image frames. The data representing the common view relationship can be used as the truth value of image retrieval for model evaluation, or as input data for training existing image retrieval models.

[0091] Based on the idea of ​​the above technical solution, the technical solution of the present invention provides a Figure 2 The method for obtaining a true retrieval value based on continuous image frames shown in the figure comprises:

[0092] Step S100, obtaining a visual map, wherein the visual map includes a map key frame and a first three-dimensional coordinate set corresponding to the map key frame.

[0093] A visual map is a series of map files including map keyframes, map keyframe image feature sets, and three-dimensional point cloud coordinate sets corresponding to these image feature sets. A visual map comes from a high-precision map file created by collecting video sequences or images within a certain spatial range using panoramic / fisheye / digital cameras and other devices. Some of the images form the map keyframes, and map image features are extracted from these map keyframes to form a keyframe image feature set. Each keyframe map feature corresponds to the three-dimensional point cloud coordinates of the image feature, thus forming a three-dimensional point cloud coordinate set of the map keyframe. The first three-dimensional coordinate set in step S100 refers to the three-dimensional point cloud coordinate set of the map keyframe.

[0094] Step S101 , collecting continuous image frames, and aligning the continuous image frames to a visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames.

[0095] Continuous image frames can also be video sequences that can be collected within the same spatial range by panoramic / fisheye / digital cameras and other devices, and each frame of these video sequences forms the continuous image frames. The second three-dimensional coordinate set in step S101 can be a three-dimensional point cloud coordinate set corresponding to the image features of the continuous image frames, and the three-dimensional point cloud coordinate set corresponding to the image features of the continuous image frames can be obtained by the following process: extracting image features from the continuous image frames to form an image feature set; matching the image feature set of the continuous image frames with the image feature set of the visual map to obtain the image feature set in the visual map image feature set that is closest to the image feature set of the continuous image frames and the three-dimensional point cloud coordinate set corresponding to these image features. These closest three-dimensional point cloud coordinate sets are the three-dimensional point cloud coordinate sets of the continuous image frames.

[0096] You can also use Figure 3 The process steps shown in the figure obtain the three-dimensional point cloud coordinate set of the continuous image frames to obtain the second three-dimensional coordinate set in step S101 in this example:

[0097] Step S200, obtaining a first relative position and posture of continuous image frames, wherein the first relative position and posture includes a first rotation angle and a first relative displacement of the continuous image frames relative to a reference image.

[0098] The relative pose P1 (i.e., the first relative pose) between all images in continuous image frames can be obtained through VIO (Visual-Inertial Odometry) algorithm or SLAM (Simultaneous Localization and Mapping) algorithm. The pose P1 is specifically the rotation parameter R1 (equivalent to the first rotation angle of step S200) and the translation parameter T1 (equivalent to the first relative displacement of step S200) of each frame in the continuous image frames relative to the first frame image.

[0099] Step S201, collecting a second relative position of the continuous image frames in the coordinate system of the visual map, wherein the second relative position includes a second rotation angle and a second relative displacement of the continuous image frames obtained by positioning the continuous image frames in the visual map.

[0100] The second relative position and posture of the continuous image frames in the coordinate system of the visual map can be acquired through the following process: feature matching is performed on the continuous image frames and the visual map; after feature matching, the image features in the visual map features that match the features of the continuous image frames are used as a matching image feature set, and a matching three-dimensional coordinate set corresponding to the matching image feature set in the first three-dimensional coordinate set is obtained; based on the matching three-dimensional coordinate set, the second relative position and posture corresponding to the continuous image frames is obtained by a PNP algorithm.

[0101] Since the second relative posture information includes angle information and position information, the second relative posture finally obtained includes the second rotation angle and the second relative displacement in step S201.

[0102] Step S202, calibrating a third relative pose of the continuous image frames in the visual map through the first relative pose and the second relative pose of the continuous image frames, wherein the third relative pose includes a third rotation angle and three relative displacements obtained by calibrating the continuous image frames in the visual map.

[0103] In step S202, based on the second relative pose of the continuous image frames, according to the first relative pose of the continuous image frames, the RANSAC algorithm can be used to re-incorporate all the continuous image frames into the visual map for sim3 transformation, and pose calibration can be performed through optimization to obtain the third relative pose. The third relative pose includes a third rotation angle and three relative displacements.

[0104] Through the sim3 transformation, the translation parameter T2 of the second relative posture is made to be s×r×T1+t. Among them, T2 is the translation parameter of the second relative posture, T1 is the translation parameter of the first relative posture, s is the scale parameter of the continuous image frames of the first posture, t is the translation parameter of the continuous image frames corresponding to the first posture, and r is the rotation parameter of the continuous image frames of the first posture. Through the sequence image frame Image1 with the first and second postures at the same time, the sequence image frame Image2 with only the first posture is calculated by the rotation r, translation t and scale s calculated by Image1 to calculate a posture of Image2, and the obtained posture is consistent with the scale in the visual map coordinate system. Then through optimization, the third posture is finally obtained. Step S203, based on the third relative posture of all continuous image frames, a retrieval map is obtained by triangulation reconstruction, and the retrieval map contains the second three-dimensional coordinate set corresponding to the continuous image frames.

[0105] Continue to refer Figure 2 , the method for obtaining a retrieval truth value based on continuous image frames further includes:

[0106] Step S102: generating a first bounding box set for the first three-dimensional coordinate set corresponding to the map key frame, wherein the first bounding box set includes a first bounding box of each first three-dimensional coordinate.

[0107] Step S103: generating a second bounding box set for the second three-dimensional coordinate set corresponding to the continuous image frames, wherein the second bounding box set includes a second bounding box of each second three-dimensional coordinate.

[0108] In step S102 and step S103, the first bounding box and the second bounding box may be AABB (Axis-aligned Bounding Box) bounding boxes. For example, the AABB bounding box may be a bounding box formed by comparing the maximum and minimum values ​​of all three-dimensional points in three directions of the three-dimensional coordinate system (XYZ coordinates) from the corresponding three-dimensional coordinates of the map key frame or the continuous image frame, and selecting six values ​​that meet the maximum and minimum values ​​in three directions of the three-dimensional coordinate system.

[0109] The first bounding box and the second bounding box may also use other bounding boxes, including but not limited to a bounding sphere (Sphere), an oriented bounding box OBB (Oriented bounding box), and a fixed direction convex hull FDH (Fixed directionshulls or k-DOP) and other methods.

[0110] Step S104: determining whether the second bounding box set has overlapping bounding boxes relative to the first bounding box set.

[0111] If the judgment result of step S104 is yes, step S105 is executed.

[0112] The determination of overlapping bounding boxes can be based on collision testing, that is, collision detection is performed for the first bounding box and the second bounding box based on the first bounding box set and the second bounding box set. Based on the collision testing, a first ratio of the overlapping bounding box to all second bounding boxes can be counted.

[0113] Step S105 , determining whether a first ratio of the overlapping bounding boxes to all second bounding boxes is greater than a first threshold.

[0114] In step S105, the first preset is the overlap ratio of the overlapping bounding boxes, and the first preset can be preset according to the system accuracy requirement.

[0115] If the determination result of step S105 is yes, steps S106 and S107 are executed.

[0116] Step S106 , counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates.

[0117] Step S107, determining whether the second ratio and the third ratio are both greater than a second threshold.

[0118] If the judgment result of step S107 is yes, then execute step S108, otherwise execute step S109.

[0119] Step S108: If the orientation angle between the continuous image frame and the map key frame is less than 90°, the map key frame is selected as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame.

[0120] Step S109, if the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold, the map key frame is selected as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame.

[0121] In the judgment process from step S102 to S109, bounding boxes are generated for the three-dimensional coordinates corresponding to the map key frame and the continuous image frame. By judging whether there is overlap between each continuous image frame and the bounding boxes corresponding to all the three-dimensional coordinates in the map key frame, a collision detection is performed first, and then the overlapping ratio of the bounding boxes with overlap is counted, that is, the first ratio. When the overlapping ratio is greater than the first threshold, the ratio of the three-dimensional coordinates in the corresponding continuous image frame and the map key frame in the overlapping area to all the corresponding first three-dimensional coordinates and second three-dimensional coordinates is counted, that is, the second ratio and the third ratio. If the second ratio and the third ratio are both greater than the second threshold, and the angle between the orientation of the continuous image frame and the map key frame is less than 90°, then for the continuous image frame, the map key frame that meets the above conditions is its retrieval true value.

[0122] However, if the number of three-dimensional coordinates of the image is small, the second ratio and the third ratio are both less than the second threshold, or the second ratio or the third ratio is less than the second threshold, resulting in false negatives in the retrieval results. At this time, the above technical solution adds position and angle judgment thresholds through the judgment process of step S110, and establishes the following conditions on the basis that the overlapping area of ​​the bounding box is greater than the first threshold: the distance between the spatial position of the continuous image frame and the spatial position of the corresponding map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the corresponding map key frame is less than the fourth threshold. If the map key frame corresponding to the continuous image frame meets this condition, then the map key frame is also the retrieval true value of the continuous image frame.

[0123] As Figure 2 A variation of the method and technical solution shown in the figure may be as follows: Figure 4 The steps shown provide a method for obtaining a retrieval truth value based on continuous image frames, which specifically includes the following steps:

[0124] Step S300, obtaining a visual map and an image key frame, wherein the image key frame includes an image pose and corresponding three-dimensional point cloud data.

[0125] Video sequences within a certain spatial range are collected through panoramic / fisheye / digital cameras and other devices, and a high-precision visual map is established. The map is recorded as basemap, and the image key frames in the map are recorded as KF.

[0126] Step S301, collecting continuous images to obtain a first relative position and posture of the continuous images, where the first relative position and posture is a rotation angle and a relative displacement of each frame of the image relative to the first frame of the image.

[0127] A video sequence is collected within the basemap space by the mobile device. The images in the video sequence are recorded as query. The relative pose P1 (first relative pose) between all images in the video sequence is obtained through VIO or SLAM. The pose P1 specifically includes the rotation R1 and translation T1 parameters of each frame in the query relative to the first frame.

[0128] Step S302, obtaining a second relative pose of the continuous image in the coordinate system of the visual map, wherein the second relative pose is the rotation angle and relative displacement of each frame of the image aligned to the visual map; and restoring the third relative pose of the continuous image in the visual map through the first relative pose and the second relative pose of the continuous image.

[0129] The query image of the video sequence is registered to the basemap using visual features through matching and PNP method, and the successfully registered query image pose P2 (second relative pose) is obtained. P2 represents the pose represented by the rotation R2 and translation T2 of the query image in the map coordinate system where the basemap is located. Based on the pose P2, all queries are re-incorporated into the basemap according to the pose P1 using the Ransac method. Furthermore, the poses of all KFs in the basemap are fixed, and the poses of the queries are optimized through bundle adjustment to obtain the final pose P3 (third relative pose) of all queries in the map coordinate system where the basemap is located.

[0130] The pose here includes position and posture. P3 consists of angle information R3 and position information T3. P1 is the relative pose of the collected continuous images, P2 is the pose of the collected continuous images successfully located in the basemap, and P3 is to restore the pose of P1 in the basemap through P2. There are two coordinate systems involved here, one is the relative coordinate system of P1, and the other is the coordinate system with real scale in the basemap. The purpose is to accurately place the collected continuous images in the basemap. Accurate placement requires accurate pose.

[0131] Step S303: triangulate the third relative pose of the continuous images to obtain a retrieval map and corresponding three-dimensional point cloud data.

[0132] All query poses P3 obtained in step S302 are triangulated to obtain a querymap, which contains all triangulated 3D point sets.

[0133] Step S304, generating a bounding box based on the three-dimensional point cloud for the three-dimensional point cloud data corresponding to the image key frame and the three-dimensional point cloud data corresponding to the retrieval map, and calculating the image space position and horizontal orientation information of the visual map and the retrieval map.

[0134] In step S304, all 3D points in each query map are obtained from the querymap, denoted as query3D, and all 3D points in each KF map are obtained from the basemap, denoted as KF3D. An AABB bounding box is generated for all query3D and KF3D points, and the position information and horizontal orientation information of each KF and query are calculated.

[0135] Step S305, performing collision detection on the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame. If the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame have overlapping bounding boxes and the first ratio of the overlapping bounding boxes is greater than a first threshold, the second ratio and the third ratio of the three-dimensional point cloud corresponding to the overlapping bounding box relative to all three-dimensional point clouds are counted.

[0136] Step S306: If the second ratio and the third ratio are both greater than the second threshold, and the orientation angle between the continuous image and the corresponding image key frame is less than 90°, the image key frame is selected as the retrieval true value of the continuous image key frame.

[0137] In steps S305 and S306, it is necessary to determine whether there is overlap between each query graph and all KF bounding boxes in the basemap, that is, first perform a collision test, and then count the overlap ratio of the bounding boxes with overlap. When the overlap ratio is greater than the threshold Th1, count the ratio of the 3D points in the corresponding query and KF to all the corresponding 3D points in the overlapping area, and require that both ratios are greater than the threshold Th2. At the same time, it is required that the angle between the two images is less than 90°. For each query graph, the KF that meets the above conditions is used as the retrieval truth value.

[0138] Step S307, if at least one of the second ratio and the third ratio is less than or equal to the second threshold and the spatial position of the continuous image and the corresponding image key frame is less than the third threshold, and the orientation angle between the continuous image and the corresponding image key frame is less than the third threshold, then the image key frame is selected as the retrieval true value of the continuous image key frame.

[0139] If the number of generated 3D points is small, the ratio of 3D points may be less than Th2, resulting in false negatives in the search results. In step S307, position and angle judgment thresholds are added. On the basis that the bounding box overlap area is greater than Th1, the KF frame whose query and KF spatial positions are less than threshold Th3 and whose angle difference is less than threshold Th4 is taken as the true value.

[0140] Step S308 : selecting the image key frames that are the true values ​​of the continuous image key frames as the co-viewing image set with the continuous image key frames.

[0141] In this embodiment, the KF (which should be a part of the key frames here) that satisfies step S306 or step S307 is taken as the true value of the query, that is, it is considered that there is a common view relationship between the KF and the query, and the common view relationship is recorded as GT.

[0142] As Figure 2 A variation of the method and technical solution shown in the figure may be as follows: Figure 5 The steps shown provide a method for obtaining a retrieval truth value based on continuous image frames, which specifically includes the following steps:

[0143] Step S400, obtaining a visual map and an image key frame, wherein the image key frame includes an image pose and corresponding three-dimensional point cloud data.

[0144] Step S401, collecting continuous images to obtain a first relative position and posture of the continuous images, where the first relative position and posture is a rotation angle and a relative displacement of each frame of the image relative to the first frame of the image.

[0145] Step S402, obtaining a second relative pose of the continuous image in the coordinate system of the visual map, wherein the second relative pose is the rotation angle and relative displacement of each frame of the image aligned to the visual map; and restoring the third relative pose of the continuous image in the visual map through the first relative pose and the second relative pose of the continuous image.

[0146] Step S403: triangulate the third relative pose of the continuous images to obtain a retrieval map and corresponding three-dimensional point cloud data.

[0147] Step S404, generating a bounding box based on the three-dimensional point cloud for the three-dimensional point cloud data corresponding to the image key frame and the three-dimensional point cloud data corresponding to the retrieval map, and calculating the image space position and horizontal orientation information of the visual map and the retrieval map.

[0148] Step S405, performing collision detection on the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame. If the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame have overlapping bounding boxes and a first ratio of the overlapping bounding boxes is greater than a first threshold, the image key frame is selected as the retrieval true value of the continuous image key frame.

[0149] Step S406 : selecting the image key frames that are the true retrieval values ​​of the continuous image key frames as the co-viewing image set with the continuous image key frames.

[0150] and Figure 4 The difference between the technical solutions shown is that Figure 5 The technical solution shown takes the KF (which should be a part of the key frame here) that satisfies the collision test in step S405 as the true value of the query, that is, it is considered that there is a common view relationship between the KF and the query, and the common view relationship is recorded as GT. The specific contents of other technical solutions shown in steps S400 to S405 can be referred to other embodiments, and will not be repeated here.

[0151] The overall process of the technical solution of the present invention may include the following: Figure 5 The method steps shown, i.e., only using the result of the bounding box collision detection (i.e., the first ratio) for true value judgment, no further judgment based on the 3D point ratio (i.e., the second ratio and the third ratio), and taking the collision detection result as the final result, should also be within the scope of protection of this patent.

[0152] As Figure 2 A variation of the method and technical solution shown in the figure may be as follows: Figure 6 The steps shown provide a method for obtaining a retrieval truth value based on continuous image frames, which specifically includes the following steps:

[0153] Step S500, obtaining a visual map and an image key frame, wherein the image key frame includes an image pose and corresponding three-dimensional point cloud data.

[0154] Step S501, collecting continuous images to obtain a first relative position and posture of the continuous images, where the first relative position and posture is a rotation angle and a relative displacement of each frame of the image relative to the first frame of the image.

[0155] Step S502, obtaining a second relative pose of the continuous image in the coordinate system of the visual map, wherein the second relative pose is the rotation angle and relative displacement of each frame of the image aligned to the visual map; and restoring the third relative pose of the continuous image in the visual map through the first relative pose and the second relative pose of the continuous image.

[0156] Step S503: triangulate the third relative pose of the continuous images to obtain a retrieval map and corresponding three-dimensional point cloud data.

[0157] Step S504, generating a bounding box based on the three-dimensional point cloud for the three-dimensional point cloud data corresponding to the image key frame and the three-dimensional point cloud data corresponding to the retrieval map, and calculating the image space position and horizontal orientation information of the visual map and the retrieval map.

[0158] Step S505, performing collision detection on the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame. If the bounding box of the three-dimensional point cloud corresponding to the retrieval map and the bounding box of the three-dimensional point cloud corresponding to the image key frame have overlapping bounding boxes and the first ratio of the overlapping bounding boxes is greater than a first threshold, then counting the second ratio and third ratio of the three-dimensional point cloud corresponding to the overlapping bounding box relative to all three-dimensional point clouds.

[0159] Step S506: if the second ratio and the third ratio are both greater than the second threshold, and the orientation angle between the continuous image and the corresponding image key frame is less than 90°, the image key frame is selected as the retrieval true value of the continuous image key frame.

[0160] Step S507 : The image key frames selected as the retrieval truth values ​​of the continuous image key frames are the co-viewed image sets with the continuous image key frames.

[0161] The difference from the technical solution shown in Figure s is that Figure 6 The technical solution shown takes the KF (which should be a part of the key frame here) that meets the condition of step S506 as the true value of the query, and considers that there is a common view relationship between the KF and the query, and the common view relationship is recorded as GT. The specific contents of other technical solutions shown in steps S500 to S507 can be referred to other embodiments, and will not be repeated here.

[0162] The embodiments of the technical solution of the present invention can also be Figure 4 Remove individually based on the process steps shown Figure 4 Step S307, that is, Figure 6 The process steps (S500 to S507) shown do not affect the purpose of obtaining the retrieval truth value of the technical solution of the present invention.

[0163] Based on the above embodiment, an image retrieval method based on continuous image frames is provided. Figure 7 As shown, the following steps may be included:

[0164] Step S600: collecting continuous image frames to be retrieved.

[0165] The continuous image frames to be retrieved are video sequences captured by panoramic / fisheye / digital cameras and other devices within the spatial range of the visual map, and each frame of the video sequence forms the continuous image frames to be retrieved.

[0166] Step S601 : obtaining a co-viewing image set corresponding to the continuous image frames by a retrieval truth acquisition method based on continuous image frames.

[0167] The specific scheme of obtaining the co-viewing image set corresponding to the continuous image frame based on the retrieval truth acquisition method of the continuous image frame can be referred to Figure 2 , Figures 4 to 6 The process steps in any diagram will not be repeated here.

[0168] Step S602: obtaining a map key frame closest to the continuous image frame through KNN query based on the global features of the continuous image frame.

[0169] In step S602, the process of retrieving the continuous image frames in the map keyframes of the visual map using global features to obtain the map keyframe closest to the continuous image frames may include the following process: extracting the first global features and the second global features of the map keyframes and the continuous image frames; and selecting the matching first global features and the map keyframes corresponding to the first global features based on the second global features of the continuous image frames, and the map keyframes are the map keyframes closest to the continuous image frames. Specifically, the matching first global features and their corresponding map keyframes may be selected based on the second global features of the continuous image frames through the KNN query algorithm. More specifically, the first global features and their corresponding map keyframes whose distance from the second global features of the continuous image frames is less than or equal to a preset selection distance may be selected as the map keyframes closest to the continuous image frames.

[0170] Further global features, namely the first global feature and the second global feature, may be image attributes extracted from the corresponding image frame as a whole, and may be at least one or more of the following features, including: color feature, texture feature and shape feature.

[0171] Step S603: If the retrieved map key frame is the retrieval truth value of the co-visible image set corresponding to the continuous image frames, it is determined that the retrieval is successful.

[0172] In other embodiments, the existing global retrieval method can be used to generate global features for all key frames in the visual map as a global feature set, and then global features are generated for all image frames to be retrieved. The global features of each image frame to be retrieved are queried in the global feature set by KNN or by selecting N map key frames with the smallest distance between the global features of the two as retrieval results, and it is determined whether the retrieval result is in the co-viewing image set (GT) corresponding to the image frame to be retrieved. If the retrieval result shows that at least one map key frame obtained by the retrieval result is in the co-viewing image set (GT) corresponding to the image frame to be retrieved, the retrieval is determined to be successful, otherwise the retrieval is failed.

[0173] The technical solution of this embodiment uses a high-precision visual map as the base map, in which all key frames are used as the database to be retrieved, and all continuous images in the video sequence collected later are used as the query map. By deeply coupling all images of the video sequence to the high-precision map, the positional relationship between the video sequence images and the database images is obtained. This positional relationship is already better than obtaining the co-visual map corresponding to the query map based on GPS. (GPS has errors, and this method has smaller deep coupling errors). Moreover, this process further optimizes the co-visual relationship between the query map and the database map based on the spatial 3D points between each key frame, so that a good true value can be obtained for retrieval.

[0174] In this technical solution, the position information of the video sequence image relative to the high-precision map can be obtained by other means, such as RTK or fusion of other sensors such as Lidar. This part requires greater hardware and labor costs than this solution. This technology only requires data collection itself, without adding other auxiliary equipment.

[0175] The technical solution of this embodiment can not only be used to determine whether the retrieval result is correct, but also can be used as data input in global feature training based on deep learning, as well as other scenarios using similar co-viewing relationships. The retrieval determination is only one of the application scenarios. The above describes the specific embodiments of the present invention.

[0176] It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which will not affect the essential content of the present invention.

Claims

1. A method for obtaining true retrieval values ​​based on continuous image frames, characterized in that: include: Acquire a visual map, wherein the visual map includes a map key frame and a first three-dimensional coordinate set corresponding to the map key frame; Acquire continuous image frames, and align the continuous image frames to a visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames; Generate a first bounding box set for the first three-dimensional coordinate set corresponding to the map key frame, wherein the first bounding box set includes a first bounding box for each first three-dimensional coordinate; generating a second bounding box set for a second three-dimensional coordinate set corresponding to the continuous image frames, the second bounding box set including a second bounding box for each second three-dimensional coordinate; If the second bounding box set has overlapping bounding boxes relative to the first bounding box set, based on a first ratio of the overlapping bounding boxes to all second bounding boxes, the corresponding map key frame is selected as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frames; The first three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the map key frame, and the second three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the continuous image frame; The second three-dimensional coordinate set corresponding to the continuous image frames is obtained by the following steps: Acquire a first relative position and posture of the continuous image frames, wherein the first relative position and posture includes a first rotation angle and a first relative displacement of the continuous image frames relative to a reference image; Collecting a second relative position of the continuous image frames in the coordinate system of the visual map, wherein the second relative position includes a second rotation angle and a second relative displacement obtained by positioning the continuous image frames in the visual map; Calibrate a third relative posture of the continuous image frames in the visual map by using the first relative posture and the second relative posture of the continuous image frames, wherein the third relative posture includes a third rotation angle and a third relative displacement obtained by calibrating the continuous image frames in the visual map; Based on the third relative positions of all the consecutive image frames, a retrieval map is obtained by triangulation reconstruction, wherein the retrieval map contains a second three-dimensional coordinate set corresponding to the consecutive image frames.

2. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The step of selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes: If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates; If the second ratio and the third ratio are both greater than the second threshold, and the orientation angle between the continuous image frame and the map key frame is less than 90°, the map key frame is selected as the retrieval truth value of the continuous image frame to obtain the co-viewed image set of the continuous image frame.

3. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The step of selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes: If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates; If the second ratio and the third ratio are both greater than a second threshold, and the orientation angle between the continuous image frame and the map key frame is less than 90°, the map key frame is selected as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame; If at least one of the second ratio and the third ratio is less than or equal to the second threshold, then when the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold, the map key frame is selected as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame.

4. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The step of selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes: If the first ratio is greater than a first threshold, counting a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all first three-dimensional coordinates, and counting a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all second three-dimensional coordinates; If at least one of the second ratio and the third ratio is less than or equal to the second threshold, then when the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold, the map key frame is selected as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame.

5. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The step of selecting the corresponding map key frame as the retrieval truth value of the continuous image frame based on the first ratio of the overlapping bounding box to all the second bounding boxes to obtain the co-viewed image set of the continuous image frame includes: If the first ratio is greater than a first threshold, the map key frame is selected as a retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frames.

6. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The acquiring of the first relative position of the continuous image frames comprises: The first relative position and posture of the continuous image frames are obtained by using a VIO algorithm or a SLAM algorithm.

7. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The step of acquiring the second relative position of the continuous image frames in the coordinate system where the visual map is located comprises: Performing feature matching between the continuous image frames and the visual map to obtain a matching three-dimensional coordinate set in a first three-dimensional coordinate set; Based on the matching three-dimensional coordinate set, a second relative posture corresponding to the continuous image frames is obtained through a PNP algorithm.

8. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The calibrating the third relative pose of the continuous image frames in the visual map by using the first relative pose and the second relative pose of the continuous image frames comprises: On the basis of the second relative pose of the continuous image frames, according to the first relative pose of the continuous image frames, all the continuous image frames are re-incorporated into the visual map for pose calibration using the RANSAC algorithm to obtain the third relative pose.

9. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: The first bounding box and the second bounding box are AABB bounding boxes, bounding spheres, directional bounding boxes or fixed-directional convex hulls.

10. The method for obtaining true retrieval value based on continuous image frames according to claim 1, characterized in that: Also includes: A collision detection is performed on the first bounding box and the second bounding box based on the first bounding box set and the second bounding box set to count a first ratio of the overlapping bounding boxes to all second bounding boxes.

11. An image retrieval method based on continuous image frames, characterized in that: include: Acquiring continuous image frames to be retrieved; Obtaining a co-viewing image set corresponding to the continuous image frames based on a retrieval truth value acquisition method based on continuous image frames according to any one of claims 1 to 10; The continuous image frames are queried by KNN based on global features to obtain the map key frame closest to the continuous image frames; If the retrieved map key frame is the retrieval truth value of the co-visual image set corresponding to the continuous image frame, the retrieval is judged to be successful.

12. The image retrieval method based on continuous image frames according to claim 11, characterized in that: The performing feature matching between the continuous image frames and the map key frames of the visual map to retrieve the map key frames closest to the continuous image frames comprises: Extracting first global features and second global features of the map key frames and continuous image frames; A matching first global feature and a map key frame corresponding to the first global feature are selected based on the second global feature of the continuous image frame, and the map key frame is the map key frame closest to the continuous image frame.

13. The image retrieval method based on continuous image frames according to claim 12, characterized in that: The selecting of the matched first global feature and the map key frame corresponding to the first global feature based on the second global feature of the continuous image frames comprises: Based on the second global features of the continuous image frames, a matching first global feature and its corresponding map key frame are selected through a KNN query algorithm.

14. The image retrieval method based on continuous image frames according to claim 12, characterized in that: The selecting of the matched first global feature and the map key frame corresponding to the first global feature based on the second global feature of the continuous image frames comprises: A first global feature and a corresponding map key frame are selected, the distance between the first global feature and the second global feature of the continuous image frames being less than or equal to a preset selection distance.

15. A retrieval truth value acquisition system based on continuous image frames, characterized in that: include: An acquisition unit, adapted to acquire a visual map, wherein the visual map comprises a map key frame and a first three-dimensional coordinate set corresponding to the map key frame; A first acquisition unit, adapted to acquire continuous image frames, and align the continuous image frames to a visual map to obtain a second three-dimensional coordinate set corresponding to the continuous image frames; A first bounding box generating unit, adapted to generate a first bounding box set for a first three-dimensional coordinate set corresponding to the map key frame, wherein the first bounding box set includes a first bounding box for each first three-dimensional coordinate; A second bounding box generating unit, adapted to generate a second bounding box set for a second three-dimensional coordinate set corresponding to the continuous image frames, the second bounding box set comprising a second bounding box for each second three-dimensional coordinate; A selection unit, adapted to select a corresponding map key frame as a retrieval truth value of the continuous image frame based on a first ratio of the overlapping bounding boxes to all the second bounding boxes when the second bounding box set has an overlapping bounding box relative to the first bounding box set, so as to obtain a co-viewing image set of the continuous image frames; The first three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the map key frame, and the second three-dimensional coordinate set contains three-dimensional point cloud data corresponding to the continuous image frame; The second three-dimensional coordinate set corresponding to the continuous image frames is obtained by the following steps: Acquire a first relative position and posture of the continuous image frames, wherein the first relative position and posture includes a first rotation angle and a first relative displacement of the continuous image frames relative to a reference image; Collecting a second relative position of the continuous image frames in the coordinate system of the visual map, wherein the second relative position includes a second rotation angle and a second relative displacement obtained by positioning the continuous image frames in the visual map; Calibrate a third relative posture of the continuous image frames in the visual map by using the first relative posture and the second relative posture of the continuous image frames, wherein the third relative posture includes a third rotation angle and a third relative displacement obtained by calibrating the continuous image frames in the visual map; Based on the third relative positions of all the consecutive image frames, a retrieval map is obtained by triangulation reconstruction, wherein the retrieval map contains a second three-dimensional coordinate set corresponding to the consecutive image frames.

16. The system for obtaining true retrieval values ​​based on continuous image frames according to claim 15, characterized in that: The selection unit comprises: a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold; The first selection subunit is adapted to select the map key frame as the retrieval true value of the continuous image frame to obtain a co-viewing image set of the continuous image frame when the second ratio and the third ratio are both greater than a second threshold and the orientation angle between the continuous image frame and the map key frame is less than 90°.

17. The system for obtaining true retrieval values ​​based on continuous image frames according to claim 15, characterized in that: The selection unit comprises: a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold; A first selection subunit is adapted to select the map key frame as the retrieval truth value of the continuous image frame to obtain a co-viewing image set of the continuous image frame when the second ratio and the third ratio are both greater than a second threshold and the orientation angle between the continuous image frame and the map key frame is less than 90°; The second selection subunit is suitable for selecting the map key frame as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when at least one of the second ratio and the third ratio is less than or equal to the second threshold, the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold.

18. The system for obtaining true retrieval values ​​based on continuous image frames according to claim 15, characterized in that: The selection unit comprises: a counting subunit adapted to count a second ratio of the first three-dimensional coordinates enclosed by the overlapping bounding boxes to all the first three-dimensional coordinates, and to count a third ratio of the second three-dimensional coordinates enclosed by the overlapping bounding boxes to all the second three-dimensional coordinates, when the first ratio is greater than a first threshold; The second selection subunit is suitable for selecting the map key frame as the retrieval true value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when at least one of the second ratio and the third ratio is less than or equal to the second threshold, the distance between the spatial position of the continuous image frame and the spatial position of the map key frame is less than the third threshold, and the orientation angle between the continuous image frame and the map key frame is less than the fourth threshold.

19. The system for obtaining true retrieval values ​​based on continuous image frames according to claim 15, characterized in that: The selection unit comprises: The third selection subunit is adapted to select the map key frame as the retrieval truth value of the continuous image frame to obtain the co-viewing image set of the continuous image frame when the first ratio is greater than a first threshold.

20. An image retrieval system based on continuous image frames, characterized in that: include: A second acquisition unit, adapted to acquire continuous image frames to be retrieved; The retrieval truth acquisition system according to any one of claims 15 to 19, adapted to obtain a co-viewing image set corresponding to the collected continuous image frames to be retrieved; A matching unit, adapted to obtain a map key frame closest to the continuous image frame through a KNN query based on the global features of the continuous image frame; The judging unit is adapted to judge that the retrieval is successful when the retrieved map key frame is the retrieval truth value of the co-visual image set corresponding to the continuous image frames.

Images