Image retrieval methods, visual positioning methods, devices and equipment in visual positioning

Abstract

This application provides an image retrieval method, visual positioning method, apparatus, and device for visual positioning. The method includes: acquiring a pose transformation matrix and a first odometer pose of a current image; wherein the first odometer pose is the pose data of the current image in an odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; converting the first odometer pose into a second odometer pose according to the pose transformation matrix; wherein the second odometer pose is the pose data of the first odometer pose in a positioning coordinate system, and the second odometer data includes second rotation data and second displacement data; determining a candidate image set from a preset positioning image set based on the second rotation data and second displacement data; and determining a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images. This results in more accurate target candidate images and improves positioning accuracy.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to an image retrieval method, visual positioning method, apparatus and device in visual positioning. Background Technology

[0002] Vision-based spatial positioning technology often uses devices equipped with cameras to capture images. By identifying environmental feature points in the images, the current position and orientation of the device can be calculated, enabling automatic recognition of the surrounding environment and providing users with an interactive visual experience.

[0003] In related technologies, servers typically use images captured by the device to perform global image retrieval in a preset database. Then, using the top N candidate images whose similarity meets the threshold obtained from the global image retrieval, the image captured by the device is matched with the candidate images to obtain a matching result image. Finally, the current position and orientation of the device are calculated based on the corresponding 3D points in the matching result image to achieve visual positioning.

[0004] However, in scenarios with high similarity, such as different waiting areas in an airport, different locations in an underground parking garage, or a park, global image retrieval often yields a large number of incorrect candidate images, resulting in significant positioning errors. Summary of the Invention

[0005] In view of the above problems, embodiments of this application provide an image retrieval method, visual positioning method, apparatus and device for visual positioning, which can obtain more accurate candidate images, thereby improving the accuracy and robustness of positioning.

[0006] In a first aspect, embodiments of this application provide an image retrieval method for visual positioning, including:

[0007] Obtain the pose transformation matrix and the first odometer pose of the current image; wherein, the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data;

[0008] According to the pose transformation matrix, the first odometer pose is transformed into a second odometer pose; wherein, the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer data includes second rotation data and second displacement data.

[0009] Based on the second rotation data and the second displacement data, a candidate image set is determined from the preset positioning image set;

[0010] A predetermined number of images with the highest similarity to the current image are determined from the candidate image set as target candidate images.

[0011] Optionally, obtaining the pose transformation matrix includes:

[0012] Obtain N historical positioning image data prior to the current time according to the time sequence; wherein each historical positioning image data includes at least a historical odometry pose and a historical visual pose; N is a natural number greater than or equal to 3;

[0013] Based on the N historical positioning image data, the pose transformation matrix between the odometer coordinate system and the positioning coordinate system is calculated.

[0014] Optionally, determining the candidate image set from the preset positioning image set based on the second rotation data and the second displacement data includes:

[0015] Obtain the positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set;

[0016] Determine a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data;

[0017] The location images whose first difference and second difference both satisfy the preset conditions are selected as images in the candidate image set.

[0018] Optionally, the step of selecting the location image whose first difference and second difference both satisfy a preset condition as the image in the candidate image set includes:

[0019] The positioning images whose first difference is less than a preset distance threshold are identified as images to be determined;

[0020] Images whose second difference is less than a preset angle threshold are identified as images in the candidate image set.

[0021] Optionally, the step of selecting the location image whose first difference and second difference both satisfy a preset condition as the image in the candidate image set includes:

[0022] The positioning images whose second difference is less than a preset angle threshold are identified as images to be determined;

[0023] Images whose first difference is less than a preset distance threshold are identified as images in the candidate image set.

[0024] Optionally, determining a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images includes:

[0025] Based on a preset algorithm, the similarity between the images in the candidate image set and the current image is determined;

[0026] Images are sorted in descending order of similarity, and a predetermined number of images with the highest similarity are selected as the target candidate images.

[0027] Optionally, the preset algorithm includes any one of the following: descriptor matching algorithm, bag-of-words model algorithm, and deep learning algorithm.

[0028] Secondly, embodiments of this application also provide a visual positioning method, including:

[0029] Get the current image;

[0030] Based on the current image, a target candidate image for positioning is determined from a preset positioning image set;

[0031] Based on the target candidate image, determine the current position and orientation of the device to be located;

[0032] The target candidate image is determined from the preset positioning image set after the first odometer pose of the current image is converted into a second odometer pose according to the obtained pose transformation matrix, and then based on the second rotation data and the second displacement data included in the second odometer pose.

[0033] Thirdly, embodiments of this application also provide an image retrieval device for visual positioning, comprising:

[0034] The acquisition module is used to acquire the pose transformation matrix and the first odometer pose of the current image; wherein, the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data;

[0035] The conversion module is used to convert the first odometer pose into a second odometer pose according to the pose conversion matrix; wherein the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer data includes second rotation data and second displacement data.

[0036] The determining module is used to determine a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data; and to determine a preset number of images with the highest similarity to the current image from the candidate image set as image retrieval results.

[0037] Fourthly, embodiments of this application also provide a visual positioning device, including:

[0038] The acquisition module is used to acquire the current image;

[0039] The determination module is used to determine, based on the current image, a target candidate image for positioning from a preset positioning image set; and to determine the current position and orientation of the device to be positioned based on the target candidate image.

[0040] The target candidate image is determined from the preset positioning image set based on the second rotation data and the second displacement data included in the second odometry pose after the first odometry pose of the current image is converted into the second odometry pose according to the obtained pose transformation matrix.

[0041] Fifthly, embodiments of this application also provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image retrieval method in visual positioning as described in any one of the first aspects of the present application; or, the processor executes the computer program to implement the steps of the visual positioning method as described in the second aspect of the present application.

[0042] In a sixth aspect, embodiments of this application also provide a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the image retrieval method in visual positioning as described in any one of the first aspects of embodiments of this application; or, when the computer program is executed by a processor, it implements the steps of the visual positioning method as described in the second aspect of embodiments of this application.

[0043] This application provides an image retrieval method, visual positioning method, apparatus, device, and computer storage medium for visual positioning. The image retrieval method for visual positioning includes: acquiring a pose transformation matrix and a first odometer pose of a current image; wherein the first odometer pose is the pose data of the current image in an odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; converting the first odometer pose into a second odometer pose according to the pose transformation matrix; wherein the second odometer pose is the pose data of the first odometer pose in a positioning coordinate system, and the second odometer data includes second rotation data and second displacement data; determining a candidate image set from a preset positioning image set based on the second rotation data and second displacement data; and determining a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images. According to the technical solution of this application, more accurate candidate images can be obtained, thereby improving the accuracy and robustness of positioning.

[0044] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of an image retrieval method in visual positioning according to an embodiment of this application.

[0047] Figure 2 This is a schematic diagram of the structure of an image retrieval device in visual positioning according to an embodiment of this application.

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0049] Figure 4 This is a flowchart of the visual positioning method according to an embodiment of this application.

[0050] Figure 5 This is a schematic diagram of the structure of the visual positioning device according to an embodiment of this application.

[0051] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims and drawings of this application are intended to cover non-exclusive inclusion.

[0054] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0056] Furthermore, the terms "first," "second," etc., in the specification and claims of this application or in the aforementioned drawings are used to distinguish different objects rather than to describe a specific order, and may explicitly or implicitly include one or more of the features.

[0057] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0058] Vision-based spatial positioning technology uses a device equipped with a camera to capture images. By identifying environmental feature points in the images, the current position and orientation of the device are calculated, enabling automatic recognition of the surrounding environment and providing users with an interactive visual experience.

[0059] In related technologies, servers typically use images captured by devices to perform global image retrieval in a pre-set database. For example, they might use a bag-of-words model or deep learning algorithms to perform global image retrieval in a known image library, obtaining the top N candidate images whose similarity meets a threshold. The image captured by the device is then matched with these candidate images to obtain a matching result image. The device's current position and orientation are then calculated based on the corresponding 3D points in the matching result image, achieving visual localization. However, global image retrieval based on the bag-of-words model or deep learning algorithms relies on image texture. Scenes with high similarity, such as airport waiting areas with uniform building structures and textures, underground parking garages lacking distinct differences, identical staircases on two floors of a shopping mall, or repetitive displays of similar plants and trees in parks or scenic areas, often contain dynamic, weak, similar, or identical textures. This makes retrieval errors highly likely, resulting in a large number of incorrect candidate images and significant localization errors.

[0060] Based on this, in order to reduce errors during image retrieval, this application provides an image retrieval method in visual positioning. First, the device's odometer data is used to greatly narrow down the candidate image range, and then a global image retrieval method is used within this candidate range to obtain more accurate candidate images, thereby improving positioning accuracy.

[0061] The image retrieval method in visual positioning according to embodiments of this application will be described in detail below with reference to the accompanying drawings and specific examples.

[0062] <First Embodiment>

[0063] <Method>

[0064] Figure 1 This is a flowchart of an image retrieval method in visual positioning according to an embodiment of this application. Specifically, the image retrieval method in visual positioning of this embodiment can be executed by an electronic device equipped with an odometer and a camera.

[0065] like Figure 1 As shown, the image retrieval method in visual positioning of this embodiment may include the following steps 1100 to 1400.

[0066] Step 1100: Obtain the pose transformation matrix and the first odometer pose of the current image; wherein, the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data.

[0067] Specifically, the pose transformation matrix is ​​used to align the odometry coordinate system to the positioning coordinate system. In this step, when obtaining the pose transformation matrix, N historical positioning image data prior to the current time can be obtained according to the time sequence; based on the N historical positioning image data, the pose transformation matrix between the odometry coordinate system and the positioning coordinate system is calculated. This pose transformation matrix includes a rotation matrix, a translation matrix, and a scale; each historical positioning image data includes at least a historical odometry pose and a historical visual pose; N is a natural number greater than or equal to 3.

[0068] In one example, both the historical odometry pose and the historical visual pose are 6DoF pose data. In this embodiment, the sim3 solution method of the Random Sample Consensus (RANSAC) algorithm is used. That is, N pairs of matching points are used to solve for the similarity transformation, and then the pose transformation matrix Trans between the two coordinate systems is solved. The pose transformation matrix Trans includes a rotation matrix R, a translation matrix t, and a scale. Among them, the rotation matrix R is a 3*3 matrix, and the translation matrix t is a 3*1 matrix.

[0069] Specifically, the electronic device can store N historical positioning image data fusionFrames from the most recent time period. Each historical positioning image data fusionFrame includes a timestamp, a historical odometry pose P1 (6Dof pose [R|t]), and a historical visual pose P2 (6Dof pose [R|t]). The stored N historical positioning image data are continuously updated; as a new frame of historical positioning image data arrives, the old frame is discarded. Reading the N historical positioning image data fusionFrames, the historical odometry pose P1 and historical visual pose P2 of each historical positioning image data fusionFrame form a pose pair. In other words, the N historical positioning image data fusionFrames provide N pose pairs.

[0070] Specifically, the N historical positioning image data fusionFrame contains N historical odometry poses P1, and the corresponding historical odometry pose P1 includes N rotation matrices R and translation matrices t, which are represented here as ([R10|t10], [R11|t11], [R12|t12], ...), where R1n is the 3*3 rotation matrix of the nth odometry pose, and t1n is the 3*1 translation matrix of the nth odometry pose.

[0071] The N historical positioning image data fusionFrame contains N historical visual poses P2, and the corresponding historical visual poses P2 include N rotation matrices R and translation matrices t, represented here as ([R20|t20], [R21|t21], [R22|t22], ...), where R2n is the 3*3 rotation matrix of the nth historical visual pose, and t2n is the 3*1 translation matrix of the nth historical visual pose. The translation matrix of the historical odometry pose P1 is p1(t10, t11, t12, ...), and the translation matrix of the historical visual pose P2 is p2(t20, t21, t22, ...). The N data points form N 3D point pairs.

[0072] The process of solving the sim3 transformation can be understood as solving the similarity transformation between two coordinate systems. Here, p2 = sR(p1) + t. After removing outliers using RANSAC, the pose transformation matrix Trans (a 4x4 matrix) of the coordinate system can be solved as long as there are more than three pairs of 3D points in the images. This matrix includes the scale parameter s, the 3x3 rotation matrix R, and the 3x1 translation matrix t.

[0073] It should be noted that the first odometry pose can be obtained by an odometry, which can be a visual-inertial odometry (VIO), a visual-inertial simultaneous localization and mapping (VISLAM) odometry, a visual odometry (VO), a simultaneous localization and mapping (SLAM) odometry, an inertial measurement unit (IMU), a global positioning system (GPS), or a method for calculating mileage based on real-time kinematic (RTK) carrier phase differential technology, etc. This embodiment does not specifically limit this method.

[0074] In this step, the current image refers to the image acquired at the current moment, the first rotation data in the first odometer pose is the first rotation matrix of the current image in the odometer coordinate system, and the first displacement data in the first odometer pose is the first translation matrix of the current image in the odometer coordinate system.

[0075] After obtaining the pose transformation matrix and the first odometry pose of the current image, step 1200 is executed.

[0076] Step 1200: Based on the pose transformation matrix, the first odometer pose is converted into the second odometer pose; wherein, the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer data includes second rotation data and second displacement data.

[0077] Specifically, the electronic device can use a pose transformation matrix to convert the first odometer pose of the current image in the odometer coordinate system into a second odometer pose in the positioning coordinate system, i.e., P3 = Trans * P1. Here, P1 is the 4*4 first odometer pose, Trans is the 4*4 pose transformation matrix, and P3 is the second odometer pose in the positioning coordinate system.

[0078] Step 1300: Determine a candidate image set from the preset positioning image set based on the second rotation data and the second displacement data.

[0079] In this step, when determining the candidate image set from the preset positioning image set, the positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set can be obtained first; then the first difference between the positioning displacement data and the second displacement data, and the second difference between the positioning rotation data and the second rotation data can be determined; finally, the positioning images whose first difference and second difference both satisfy the preset conditions are taken as images in the candidate image set.

[0080] In one feasible implementation, when an electronic device selects a positioning image whose first difference and second difference both satisfy a preset condition as an image in a candidate image set, it can first determine the positioning image whose first difference is less than a preset distance threshold as the image to be determined; and then determine the image to be determined whose second difference is less than a preset angle threshold as the image in the candidate image set.

[0081] In another feasible implementation, when the electronic device selects the positioning image whose first difference and second difference both meet the preset conditions as the image in the candidate image set, it can first determine the positioning image whose second difference is less than the preset angle threshold as the image to be determined; and then determine the image to be determined whose first difference is less than the preset distance threshold as the image in the candidate image set.

[0082] For example, suppose the preset positioning image set contains M positioning images, and the preset positioning image set is denoted as C0. Each positioning image has corresponding 6DoF pose data, that is, 3DoF positioning rotation data and 3DoF positioning displacement data.

[0083] The electronic device can first identify the positioning images in C0 whose first difference between the positioning displacement data and the second displacement data of the current image is less than a preset distance threshold, and place them in image set C1; then, identify the positioning images in image set C1 whose second difference between the positioning rotation data and the second rotation data of the current image is less than a preset angle threshold, and place them in candidate image set C2. Alternatively, the electronic device can first identify the positioning images in C0 whose second difference between the positioning rotation data and the second rotation data of the current image is less than a preset angle threshold, and place them in image set C1; then, identify the positioning images in image set C1 whose first difference between the positioning displacement data and the second displacement data of the current image is less than a preset distance threshold, and place them in candidate image set C2.

[0084] As can be seen, in this step, one can either first use displacement data to filter out the image to be determined from the preset positioning image set, and then use rotation data to filter out candidate images from the image to be determined, or first use rotation data to filter out the image to be determined from the preset positioning image set, and then use displacement data to filter out candidate images from the image to be determined. In this way, by using odometry pose as a priori, more accurate candidate images can be provided for the next positioning, thereby refining the image retrieval accuracy to a very small range and reducing the error caused by the global retrieval algorithm.

[0085] After the candidate image set is determined, step 1400 is executed.

[0086] Step 1400: Determine a preset number of images from the candidate image set that have the highest similarity to the current image as target candidate images.

[0087] In this step, the electronic device can determine the similarity between the images in the candidate image set and the current image based on a preset algorithm; sort them according to the similarity from high to low, and determine the preset number of images with the highest similarity as target candidate images. The preset algorithm can be any one of descriptor matching algorithm, bag-of-words model algorithm, and deep learning algorithm.

[0088] Continuing the example above, the electronic device can sort the candidate images in the candidate image set C2 by similarity with the current image. Specifically, it can use descriptor matching algorithms, including but not limited to ORB, SIFT, SuperPoint, etc., bag-of-words model algorithms, including but not limited to DBoW, or deep learning algorithms, including but not limited to NetVLAD, GeM to search in the candidate image set C2 and obtain the top K images with the highest similarity as the K target candidate images.

[0089] The image retrieval method in visual positioning of this embodiment obtains a pose transformation matrix and a first odometry pose of the current image. The first odometry pose is the pose data of the current image in the odometry coordinate system, including first rotation data and first displacement data. Based on the pose transformation matrix, the first odometry pose is converted into a second odometry pose. The second odometry pose is the pose data of the first odometry pose in the positioning coordinate system, including second rotation data and second displacement data. Based on the second rotation data and second displacement data, a candidate image set is determined from a preset positioning image set. From the candidate image set, a preset number of images with the highest similarity to the current image are determined as target candidate images. By using the odometry pose as a priori, more accurate candidate images can be provided for the next positioning, thereby refining the image retrieval accuracy to a very small range and reducing the error caused by the global retrieval algorithm. Furthermore, since the alignment of the odometry pose and the visual pose uses the RANSAC sim3 estimation algorithm after removing outliers, positioning pose data with large errors can be removed, and the problem of scale inconsistency between the odometry pose and the visual pose can be solved.

[0090] <device>

[0091] Figure 2 This is a schematic diagram of the structure of an image retrieval device in visual positioning according to an embodiment of this application. For example... Figure 2 As shown, the image retrieval device 2000 in the visual positioning of this embodiment may include: an acquisition module 2100, a conversion module 2200, and a determination module 2300.

[0092] The acquisition module 2100 is used to acquire the pose transformation matrix and the first odometer pose of the current image; wherein the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data.

[0093] The conversion module 2200 is used to convert the first odometer pose into the second odometer pose according to the pose conversion matrix; wherein, the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer data includes second rotation data and second displacement data.

[0094] The determining module 2300 is used to determine a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data; and to determine a preset number of images with the highest similarity to the current image from the candidate image set as the image retrieval results.

[0095] In one embodiment, the acquisition module 2100 can be specifically used to: acquire N historical positioning image data prior to the current time according to the time sequence; wherein each historical positioning image data includes at least historical odometry pose and historical visual pose; N is a natural number greater than or equal to 3; and calculate the pose transformation matrix between the odometry coordinate system and the positioning coordinate system based on the N historical positioning image data.

[0096] In one embodiment, the determining module 2300 may be specifically used to: obtain positioning rotation data and positioning displacement data corresponding to each positioning image in a preset positioning image set; determine a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data; and take the positioning image whose first difference and the second difference both satisfy the preset conditions as the image in the candidate image set.

[0097] In one embodiment, when determining a positioning image whose first difference and second difference both satisfy a preset condition as an image in the candidate image set, the determining module 2300 may specifically be used to: determine a positioning image whose first difference is less than a preset distance threshold as an image to be determined; and determine an image to be determined whose second difference is less than a preset angle threshold as an image in the candidate image set.

[0098] In one embodiment, when determining a positioning image whose first difference and second difference both satisfy a preset condition as an image in the candidate image set, the determining module 2300 may specifically be used to: determine a positioning image whose second difference is less than a preset angle threshold as an image to be determined; and determine an image to be determined whose first difference is less than a preset distance threshold as an image in the candidate image set.

[0099] In one embodiment, the determining module 2300 is specifically used to: determine the similarity between images in the candidate image set and the current image based on a preset algorithm; sort the images according to their similarity from high to low, and determine a preset number of images with the highest similarity as target candidate images. The preset algorithm includes any one of a descriptor matching algorithm, a bag-of-words model algorithm, and a deep learning algorithm.

[0100] The image retrieval device in the visual positioning of this embodiment can be used to perform the above-described... Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0101] <Electronic Devices>

[0102] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 3As shown, the electronic device 3000 of this embodiment may include: a memory 3100 and a processor 3200. The memory 3100 stores a computer program, and when the processor 3200 executes the computer program, it implements the steps of the image retrieval method in visual positioning as described in the above method embodiment.

[0103] Computer storage media

[0104] This application also provides a computer storage medium. Specifically, the computer storage medium of this embodiment stores a computer program, which, when executed by a processor, implements the steps of the image retrieval method in visual positioning as described in the above method embodiments.

[0105] <Second Embodiment>

[0106] <Method>

[0107] Figure 4 This is a flowchart of a visual positioning method according to an embodiment of this application. Specifically, the visual positioning method of this embodiment can be executed by an electronic device equipped with an odometer and a camera.

[0108] like Figure 4 As shown, the visual positioning method of this embodiment may include the following steps 4100 to 4300.

[0109] Step 4100: Obtain the current image.

[0110] The current image refers to the image acquired by the camera device at the current moment.

[0111] Step 4200: Based on the current image, determine the target candidate image for positioning from the preset positioning image set.

[0112] The target candidate image is determined from a preset positioning image set after converting the first odometry pose of the current image into a second odometry pose based on the acquired pose transformation matrix, and then using the second rotation data and second displacement data included in the second odometry pose. The specific determination method can be referred to as described above. Figure 1 The description of the image retrieval method in visual positioning shown will not be repeated here.

[0113] Step 4300: Determine the current position and orientation of the device to be located based on the target candidate image.

[0114] In this step, the electronic device can match the target candidate image with the current image to obtain the matching result image, and then solve the current position and attitude of the device based on the corresponding 3D points in the matching result image to achieve visual positioning.

[0115] The visual positioning method in this embodiment can use the current image and the odometry pose as a priori to determine the target candidate image for positioning from the preset positioning image set. This can refine the image retrieval accuracy to a very small range, reduce the error caused by the global retrieval algorithm, and thus provide a good environment for computation, effectively improving the success rate and accuracy of visual positioning.

[0116] <device>

[0117] Figure 5 This is a schematic diagram of the structure of a visual positioning device according to an embodiment of this application. Figure 5 As shown, the visual positioning device 5000 of this embodiment may include: an acquisition module 5100 and a determination module 5200.

[0118] The acquisition module 5100 is used to acquire the current image.

[0119] The determination module 5200 is used to determine a target candidate image for positioning from a preset positioning image set based on the current image; and to determine the current position and orientation of the device to be positioned based on the target candidate image.

[0120] The target candidate image is determined from a preset positioning image set based on the second rotation data and second displacement data included in the second odometry pose after the first odometry pose of the current image is converted into the second odometry pose according to the obtained pose transformation matrix.

[0121] The visual positioning device in this embodiment can be used to perform the above-described... Figure 4 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0122] <Electronic Devices>

[0123] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 6 As shown, the electronic device 6000 of this embodiment may include: a memory 6100 and a processor 6200. The memory 6100 stores a computer program, and the processor 6200 executes the computer program to implement the steps of the visual positioning method as described in the above method embodiment.

[0124] Computer storage media

[0125] This application also provides a computer storage medium. Specifically, the computer-readable storage medium of this embodiment stores a computer program, which, when executed by a processor, implements the steps of the visual positioning method as described in the above method embodiments.

[0126] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" as described in this application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several units of these means may be embodied by the same item of hardware. The use of "first," "second," and "third," etc., does not indicate any order and these words should be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

[0129] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image retrieval method in visual positioning, characterized in that, include: Obtain the pose transformation matrix and the first odometer pose of the current image; wherein, the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; According to the pose transformation matrix, the first odometer pose is transformed into the second odometer pose; wherein, the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer pose includes second rotation data and second displacement data. Based on the second rotation data and the second displacement data, a candidate image set is determined from the preset positioning image set; From the candidate image set, a predetermined number of images with the highest similarity to the current image are determined as target candidate images; The step of determining a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data includes: Obtain the positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set; Determine a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data; The location images whose first difference and second difference both satisfy the preset conditions are selected as images in the candidate image set.

2. The method according to claim 1, characterized in that, The acquisition of the pose transformation matrix includes: Obtain N historical positioning image data prior to the current time according to the time sequence; wherein each historical positioning image data includes at least a historical odometry pose and a historical visual pose; N is a natural number greater than or equal to 3; Based on the N historical positioning image data, the pose transformation matrix between the odometer coordinate system and the positioning coordinate system is calculated.

3. The method according to claim 1, characterized in that, The step of selecting the location image whose first difference and second difference both satisfy a preset condition as the image in the candidate image set includes: The positioning images whose first difference is less than a preset distance threshold are identified as images to be determined; Images whose second difference is less than a preset angle threshold are identified as images in the candidate image set.

4. The method according to claim 1, characterized in that, The step of selecting the location image whose first difference and second difference both satisfy a preset condition as the image in the candidate image set includes: The positioning images whose second difference is less than a preset angle threshold are identified as images to be determined; Images whose first difference is less than a preset distance threshold are identified as images in the candidate image set.

5. The method according to claim 1, characterized in that, The step of determining a predetermined number of images with the highest similarity to the current image from the candidate image set as target candidate images includes: Based on a preset algorithm, the similarity between the images in the candidate image set and the current image is determined; Images are sorted in descending order of similarity, and a predetermined number of images with the highest similarity are selected as the target candidate images.

6. The method according to claim 5, characterized in that, The preset algorithm includes any one of the following: descriptor matching algorithm, bag-of-words model algorithm, and deep learning algorithm.

7. A visual positioning method, characterized in that, include: Get the current image; Based on the current image, a target candidate image for positioning is determined from a preset positioning image set; Based on the target candidate image, determine the current position and orientation of the device to be located; The step of determining the target candidate image includes: acquiring a pose transformation matrix and a first odometer pose of the current image; wherein the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; converting the first odometer pose into a second odometer pose according to the pose transformation matrix; wherein the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer pose includes second rotation data and second displacement data; determining a candidate image set from a preset positioning image set according to the second rotation data and the second displacement data; and determining a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images. The step of determining a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data includes: obtaining positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set; determining a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data; and selecting positioning images whose first difference and second difference both satisfy preset conditions as images in the candidate image set.

8. An image retrieval device for visual positioning, characterized in that, include: The acquisition module is used to acquire the pose transformation matrix and the first odometer pose of the current image; wherein, the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; The conversion module is used to convert the first odometer pose into a second odometer pose according to the pose conversion matrix; wherein the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer pose includes second rotation data and second displacement data. The determining module is used to determine a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data; and to determine a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images. Specifically, the determining module is used to acquire positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set; determine a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data; and select positioning images whose first difference and second difference both satisfy preset conditions as images in the candidate image set.

9. A visual positioning device, characterized in that, include: The acquisition module is used to acquire the current image; The determining module is used to determine a target candidate image for positioning from a preset positioning image set based on the current image; Based on the target candidate image, determine the current position and orientation of the device to be located; The step of determining the target candidate image includes: acquiring a pose transformation matrix and a first odometer pose of the current image; wherein the first odometer pose is the pose data of the current image in the odometer coordinate system; the first odometer pose includes first rotation data and first displacement data; converting the first odometer pose into a second odometer pose according to the pose transformation matrix; wherein the second odometer pose is the pose data of the first odometer pose in the positioning coordinate system, and the second odometer pose includes second rotation data and second displacement data; determining a candidate image set from a preset positioning image set according to the second rotation data and the second displacement data; and determining a preset number of images with the highest similarity to the current image from the candidate image set as target candidate images. Specifically, when determining a candidate image set from a preset positioning image set based on the second rotation data and the second displacement data, the determining module is configured to: acquire positioning rotation data and positioning displacement data corresponding to each positioning image in the preset positioning image set; determine a first difference between the positioning displacement data and the second displacement data, and a second difference between the positioning rotation data and the second rotation data; and select positioning images whose first difference and second difference both satisfy preset conditions as images in the candidate image set.

10. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image retrieval method in visual positioning as described in any one of claims 1 to 6; or, the processor executes the computer program to implement the steps of the visual positioning method as described in claim 7.

11. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image retrieval method in visual positioning as described in any one of claims 1 to 6; or, when executed by a processor, the computer program implements the steps of the visual positioning method as described in claim 7.

Images