Visual repositioning method and device, equipment and storage medium
By combining data from lidar, image sensors, and IMU inertial sensors, a visual positioning map is constructed and word vectors and feature points are matched, solving the problems of increased positioning error and mismatch in existing technologies, and achieving high-precision visual repositioning.
Patent Information
- Application Number
- CN202310519500.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-12
AI Technical Summary
Existing localization methods based on algorithms such as SLAM suffer from increased localization errors as the duration and distance of movement increase. Laser point cloud relocalization schemes are prone to mismatches, while image-based relocalization schemes suffer from problems such as map skew, ghosting, and long mapping time in medium and large-scale scenes.
By combining lidar sensors, image sensors, and IMU inertial sensors, the feature information of the current frame image is compared with a pre-built visual positioning map. Relocalization is performed using word vector matching and feature point matching relationships to construct a visual positioning map, avoiding map skew and mismatches.
It improves the accuracy of visual positioning maps and the precision of repositioning, avoids problems such as map skew, ghosting, and long mapping time, and enhances the accuracy of repositioning in medium and large-scale scenarios.
Smart Images

Figure CN118941623A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to a visual relocalization method, device, equipment and storage medium. Background Art
[0002] In applications such as AR (Augmented Reality), VR (Virtual Reality), and robots, positioning is one of its basic functions. Related technologies mainly use sensors such as cameras, GPS (Global Positioning System), IMU (inertial measurement unit), lidar, and depth cameras to locate devices, that is, to calculate the posture of the device in the environment through sensor data and algorithms such as SLAM (Simultaneous Localization and Mapping).
[0003] However, a major disadvantage of positioning based on algorithms such as SLAM is that the positioning error will increase with the increase of movement time and distance. In order to solve the problem of positioning precision and accuracy, visual repositioning technology came into being. There are usually two traditional repositioning schemes: one is the repositioning scheme using laser point cloud. Since the laser point cloud has no obvious features, the similarity between each frame of laser point cloud and the map point cloud is extremely large, which leads to the problem of mismatch, which in turn leads to inaccurate repositioning. The other is an image-based repositioning scheme. Since this method relies heavily on the accuracy of visual map reconstruction, in medium and large scenes, visual map reconstruction is prone to map skew, ghosting, errors, and long map construction time, which leads to inaccurate repositioning. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the embodiments of the present disclosure provide a visual relocalization method, apparatus, device and storage medium to solve the defects in the related art.
[0005] According to a first aspect of an embodiment of the present disclosure, a visual relocalization method is provided, which is applied to a mobile device having a laser radar sensor, an image sensor, and an IMU inertial sensor, and the method includes:
[0006] In response to collecting a current frame image based on the image sensor, acquiring first feature information of the current frame image;
[0007] Comparing the first feature information with second feature information of each frame of environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected by the laser radar sensor, the image sensor, and the IMU inertial sensor;
[0008] The mobile device is relocated based on the comparison result.
[0009] In some embodiments, the first feature information includes a current image word vector of the current frame image, and the second feature information includes an environment image word vector of the environment image;
[0010] The obtaining of the first feature information of the current frame image includes:
[0011] Performing feature extraction on the current frame image to obtain feature points of the current frame image;
[0012] Inputting the feature points into a pre-trained bag-of-words model to obtain the current image word vector;
[0013] The step of comparing the first feature information with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result includes:
[0014] Determine the vector similarity between the current image word vector and the environment image word vector.
[0015] In some embodiments, relocating the mobile device based on the comparison result includes:
[0016] Selecting at least one candidate image from each frame of the environment image based on the vector similarity;
[0017] The mobile device is repositioned based on the at least one candidate image frame.
[0018] In some embodiments, relocating the mobile device based on the at least one candidate image frame includes:
[0019] Determine a feature point matching relationship between the current frame image and each candidate image in the at least one candidate image frame;
[0020] Based on the feature point matching relationship, determine the relative posture corresponding to each of the at least one frame of candidate images, where the relative posture is the posture of the current frame image relative to the visual positioning map;
[0021] Converting the relative posture based on an external parameter to obtain a candidate radar frame posture, wherein the external parameter is a relative external parameter between the laser radar sensor and the image sensor;
[0022] In response to the existence of a target radar frame posture satisfying a set condition in the candidate radar frame postures, the mobile device is repositioned based on the target radar frame posture.
[0023] In some embodiments, the method further comprises:
[0024] Taking the candidate radar frame posture as the posture at the previous moment, matching the current frame image with the map;
[0025] In response to the current frame image successfully matching the map, determining that the candidate radar frame posture meets the set condition, and using the candidate radar frame posture as the target radar frame posture;
[0026] The current frame image successfully matches the map, including:
[0027] The matching error between the point cloud of the current frame image and the point cloud of the environment image in the map is less than or equal to a preset error threshold, and the number of successfully matched points is greater than or equal to a set ratio.
[0028] In some embodiments, the method further comprises constructing the visual positioning map in advance based on:
[0029] Based on the laser radar sensor, the image sensor and the IMU inertial sensor, respectively collect the laser point cloud, the environment image sequence and the inertial data;
[0030] Determine the device posture of the mobile device using a laser synchronous positioning and mapping algorithm based on the laser point cloud and the inertial data;
[0031] A visual positioning map of the current environment is constructed based on the device posture, the laser point cloud and the environment image sequence.
[0032] In some embodiments, constructing a visual positioning map of the current environment based on the device posture, the laser point cloud, and the environment image sequence includes:
[0033] Projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point, wherein the external parameter is a relative external parameter of the laser radar sensor and the image sensor;
[0034] Determine an environment image word vector for each frame of the environment image in the environment image sequence based on the projection point;
[0035] The visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
[0036] In some embodiments, the method further comprises:
[0037] Performing voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud;
[0038] The laser point cloud is replaced based on the filtered point cloud, and the operation of projecting the laser point cloud onto a corresponding environmental image in the environmental image sequence to obtain a projection point is performed.
[0039] In some embodiments, projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point includes:
[0040] Dividing each frame of the environmental image in the environmental image sequence into a plurality of grids based on a preset grid size;
[0041] Projecting a current laser point in the laser point cloud onto a corresponding grid among a plurality of grids of the corresponding environment image based on the internal and external parameters of the image sensor to form a latest projection point, and counting the current number of projection points on the corresponding grid;
[0042] In response to detecting that the current number is less than or equal to a preset number threshold, retaining the latest projection point.
[0043] In some embodiments, the method further comprises:
[0044] In response to detecting that the current number is greater than the preset number threshold, deleting the latest projection point.
[0045] In some embodiments, determining the environment image word vector of each frame of the environment image in the environment image sequence based on the projection point includes:
[0046] Determine, based on the projection points, key point descriptors at each projection position of each frame of the environment image in the environment image sequence;
[0047] The environment image word vector is determined based on the key point descriptor.
[0048] According to a second aspect of an embodiment of the present disclosure, a visual relocalization device is provided, which is applied to a mobile device having a laser radar sensor, an image sensor, and an IMU inertial sensor, and the device includes:
[0049] An information acquisition module, configured to acquire first feature information of a current frame image in response to the current frame image being acquired by the image sensor;
[0050] A result acquisition module is used to compare the first feature information with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on the pre-collected current environment data, and the current environment data includes the laser point cloud, the environment image sequence and the inertial data of the current environment respectively collected based on the laser radar sensor, the image sensor and the IMU inertial sensor;
[0051] A relocation module is used to relocate the mobile device based on the comparison result.
[0052] In some embodiments, the first feature information includes a current image word vector of the current frame image, and the second feature information includes an environment image word vector of the environment image.
[0053] The information acquisition module comprises:
[0054] A feature point acquisition unit, used for extracting features from the current frame image to obtain feature points of the current frame image;
[0055] A word vector acquisition unit, used for inputting the feature point into a pre-trained bag-of-words model to obtain the current image word vector;
[0056] The result acquisition module is also used to determine the vector similarity between the current image word vector and the environment image word vector.
[0057] In some embodiments, the relocation module includes:
[0058] An image selection unit, configured to select at least one candidate image from each frame of the environment image based on the vector similarity;
[0059] A repositioning unit is used to reposition the mobile device based on the at least one frame of candidate image.
[0060] In some embodiments, the relocation unit is further configured to:
[0061] Determine a feature point matching relationship between the current frame image and each candidate image in the at least one candidate image frame;
[0062] Based on the feature point matching relationship, determine the relative posture corresponding to each of the at least one frame of candidate images, where the relative posture is the posture of the current frame image relative to the visual positioning map;
[0063] Converting the relative posture based on an external parameter to obtain a candidate radar frame posture, wherein the external parameter is a relative external parameter between the laser radar sensor and the image sensor;
[0064] In response to the existence of a target radar frame posture satisfying a set condition in the candidate radar frame postures, the mobile device is repositioned based on the target radar frame posture.
[0065] In some embodiments, the relocation unit is further configured to:
[0066] Taking the candidate radar frame posture as the posture at the previous moment, matching the current frame image with the map;
[0067] In response to the current frame image successfully matching the map, determining that the candidate radar frame posture meets the set condition, and using the candidate radar frame posture as the target radar frame posture;
[0068] The current frame image successfully matches the map, including:
[0069] The matching error between the point cloud of the current frame image and the point cloud of the environment image in the map is less than or equal to a preset error threshold, and the number of successfully matched points is greater than or equal to a set ratio.
[0070] In some embodiments, the apparatus further comprises a map construction module;
[0071] The map construction module includes:
[0072] A data acquisition unit, configured to respectively acquire the laser point cloud, the environment image sequence, and the inertial data based on the laser radar sensor, the image sensor, and the IMU inertial sensor;
[0073] A posture determination unit, configured to determine the device posture of the mobile device using a laser synchronous positioning and mapping algorithm based on the laser point cloud and the inertial data;
[0074] A map construction unit is used to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence.
[0075] In some embodiments, the map construction unit is further configured to:
[0076] Projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point, wherein the external parameter is a relative external parameter of the laser radar sensor and the image sensor;
[0077] Determine an environment image word vector for each frame of the environment image in the environment image sequence based on the projection point;
[0078] The visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
[0079] In some embodiments, the map construction unit is further configured to:
[0080] Performing voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud;
[0081] The laser point cloud is replaced based on the filtered point cloud, and the operation of projecting the laser point cloud onto a corresponding environmental image in the environmental image sequence to obtain a projection point is performed.
[0082] In some embodiments, the map construction unit is further configured to:
[0083] Dividing each frame of the environmental image in the environmental image sequence into a plurality of grids based on a preset grid size;
[0084] Projecting a current laser point in the laser point cloud onto a corresponding grid among a plurality of grids of the corresponding environment image based on the internal and external parameters of the image sensor to form a latest projection point, and counting the current number of projection points on the corresponding grid;
[0085] In response to detecting that the current number is less than or equal to a preset number threshold, retaining the latest projection point.
[0086] In some embodiments, the map construction unit is further configured to:
[0087] In response to detecting that the current number is greater than the preset number threshold, deleting the latest projection point.
[0088] In some embodiments, the map construction unit is further configured to:
[0089] Determine, based on the projection points, key point descriptors at each projection position of each frame of the environment image in the environment image sequence;
[0090] The environment image word vector is determined based on the key point descriptor.
[0091] According to a third aspect of an embodiment of the present disclosure, a mobile device is provided, the device comprising:
[0092] LiDAR sensor, image sensor, IMU inertial sensor, processor and memory for storing computer programs;
[0093] Wherein, the processor is configured to implement, when executing the computer program:
[0094] In response to collecting a current frame image based on the image sensor, acquiring first feature information of the current frame image;
[0095] Comparing the first feature information with second feature information of each frame of environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected by the laser radar sensor, the image sensor, and the IMU inertial sensor;
[0096] The mobile device is relocated based on the comparison result.
[0097] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the following is implemented:
[0098] In response to collecting a current frame image based on an image sensor, acquiring first feature information of the current frame image;
[0099] Comparing the first feature information with second feature information of each frame of the environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected based on a laser radar sensor, the image sensor, and an IMU inertial sensor;
[0100] The mobile device is relocated based on the comparison result.
[0101] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0102] The present disclosure obtains first feature information of the current frame image in response to the current frame image collected by the image sensor, and compares the first feature information with second feature information of each frame environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on the pre-collected current environment data, and the current environment data includes the laser point cloud, environment image sequence and inertial data of the current environment collected respectively by the lidar sensor, the image sensor and the IMU inertial sensor, and then the mobile device is relocated based on the comparison result. Since the visual positioning map is constructed based on the laser point cloud, environment image sequence and inertial data of the current environment in advance, the mobile device is relocated based on the comparison result. Compared with the repositioning scheme based only on images in the related art, the visual positioning map can avoid problems such as map skew, ghosting, errors and long map construction time, and can improve the accuracy of constructing the visual positioning map, thereby improving the accuracy of subsequent repositioning based on the visual positioning map; on the other hand, since the first feature information of the current frame image collected by the image sensor is compared with the second feature information of each frame of the environmental image in the visual positioning map, repositioning can be performed based on the image features of the current frame image and each frame of the environmental image in the visual positioning map. Compared with the repositioning scheme based only on laser point clouds in the related art, the mismatch problem caused by the lack of obvious features of the laser point clouds can be avoided, thereby improving the accuracy of repositioning.
[0103] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0104] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0105] Figure 1 is a flow chart of a visual relocalization method according to an exemplary embodiment of the present disclosure;
[0106] Figure 2 is a flowchart showing how to obtain the first feature information of the current frame image according to an exemplary embodiment of the present disclosure;
[0107] Figure 3 is a flowchart showing how to relocate the mobile device based on the comparison result according to an exemplary embodiment of the present disclosure;
[0108] Figure 4 is a flowchart showing how to relocate the mobile device based on the at least one frame of candidate image according to an exemplary embodiment of the present disclosure;
[0109] Figure 5 is a flowchart showing how to determine whether there is a target radar frame posture that meets the set conditions according to an exemplary embodiment of the present disclosure;
[0110] Figure 6 is a flowchart showing how to construct the visual positioning map according to an exemplary embodiment of the present disclosure;
[0111] Figure 7 is a flowchart showing how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence according to an exemplary embodiment of the present disclosure;
[0112] Figure 8 is a flowchart showing how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence according to another exemplary embodiment of the present disclosure;
[0113] Fig. 9 is a flowchart showing how to project the laser point cloud onto the corresponding environment image in the environment image sequence to obtain projection points based on the intrinsic parameters and extrinsic parameters of the image sensor according to an exemplary embodiment of the present disclosure;
[0114] Fig.10 is a flowchart showing how to determine the environmental image word vector of each frame of the environmental image in the environmental image sequence based on the projection point according to an exemplary embodiment of the present disclosure;
[0115] Fig.11 is a block diagram of a visual relocalization device according to an exemplary embodiment of the present disclosure;
[0116] Fig.12 is a block diagram of another visual relocalization device according to an exemplary embodiment of the present disclosure;
[0117] Fig.13 It is a block diagram of a mobile device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0118] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0119] Figure 1It is a flowchart of a visual relocalization method according to an exemplary embodiment; the method of this embodiment can be applied to a mobile device (such as a robot, an autonomous driving vehicle or a vehicle-mounted terminal, etc.) having a lidar sensor, an image sensor and an IMU inertial sensor.
[0120] like Figure 1 As shown, the method includes the following steps S101-S103:
[0121] In step S101, in response to capturing a current frame image based on the image sensor, first feature information of the current frame image is acquired.
[0122] In this embodiment, the mobile device may acquire the first feature information of the current frame image in response to capturing the current frame image based on the image sensor.
[0123] For example, when a mobile device is moving in the current environment, the mobile device can use its own image sensor (such as a camera module, etc.) to capture the current frame image, and then obtain the first feature information of the current frame image based on feature extraction and other methods.
[0124] In some embodiments, the content of the first feature information can be set based on actual business needs, such as being set to feature points and / or word vectors of the image that can reflect the difference between the current frame image and other images. This embodiment does not limit this.
[0125] In some other embodiments, the above method of obtaining the first feature information of the current frame image can also refer to the following Figure 2 The illustrated embodiment will not be described in detail here.
[0126] In step S102, the first feature information is compared with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result.
[0127] In this embodiment, after the first feature information of the current frame image is obtained, the first feature information can be compared with the second feature information of each frame environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result.
[0128] The visual positioning map can be a three-dimensional visual map for positioning constructed based on pre-collected current environment data, and the current environment data can include laser point clouds, environment image sequences, and inertial data of the current environment collected by the lidar sensor, the image sensor, and the IMU inertial sensor, respectively. It can be understood that the environment image sequence contains multiple frames of environment images, and each frame of the environment image can correspond to a posture of the mobile device.
[0129] In some embodiments, the comparison result between the first feature information and the second feature information may include the similarity between the two.
[0130] It is worth noting that the above-mentioned visual positioning map is pre-constructed based on the laser point cloud, environmental image sequence and inertial data of the current environment. That is to say, before relocating the mobile device, the mobile device can enter the current environment in advance, so as to collect the laser point cloud, environmental image sequence and inertial data of the current environment based on the laser radar sensor, the image sensor and the IMU inertial sensor carried, and construct a visual positioning map based on these data, so as to realize the construction of the visual positioning map by combining the laser point cloud data and the image information. Compared with the repositioning scheme based only on images in the related technology (i.e., using images for visual three-dimensional reconstruction, and then repositioning according to the reconstructed image point cloud, posture and image information), it can avoid the problems of visual map skewness, ghosting, errors and long map construction time, so as to improve the accuracy of constructing the visual positioning map, and on this basis, improve the accuracy of subsequent repositioning based on the visual positioning map.
[0131] In other embodiments, the construction method of the above-mentioned visual positioning map can also refer to the following Figure 6 The illustrated embodiment will not be described in detail here.
[0132] In step S103, the mobile device is relocated based on the comparison result.
[0133] In this embodiment, after the first feature information is compared with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment and a comparison result is obtained, the mobile device can be repositioned based on the comparison result.
[0134] For example, after obtaining the comparison result of the first feature information and the second feature information, an image that meets the preset conditions can be selected from each frame of the environmental image based on the comparison result, and then the current posture of the mobile device can be re-determined based on the selected image, thereby achieving repositioning of the mobile device.
[0135] In some other embodiments, the above-mentioned method of relocating the mobile device based on the comparison result can also refer to the following Figure 3 The illustrated embodiment will not be described in detail here.
[0136] As can be seen from the above description, the method of this embodiment obtains the first feature information of the current frame image in response to the current frame image collected by the image sensor, and compares the first feature information with the second feature information of each frame environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on the pre-collected current environment data, and the current environment data includes the laser point cloud, environment image sequence and inertial data of the current environment collected respectively based on the lidar sensor, the image sensor and the IMU inertial sensor, and then the mobile device is relocated based on the comparison result. Compared with the repositioning scheme based only on images in the related art, the visual positioning map is constructed based on the characteristic data. Problems such as map skewness, ghosting, errors and long map construction time can be avoided, and the accuracy of constructing the visual positioning map can be improved, thereby improving the accuracy of subsequent repositioning based on the visual positioning map. On the other hand, since the first feature information of the current frame image collected by the image sensor is compared with the second feature information of each frame of the environmental image in the visual positioning map, repositioning can be performed based on the image features of the current frame image and each frame of the environmental image in the visual positioning map. Compared with the repositioning scheme based only on laser point clouds in the related art, the mismatch problem caused by the lack of obvious features of the laser point clouds can be avoided, thereby improving the accuracy of repositioning.
[0137] Figure 2 It is a flowchart showing how to obtain the first characteristic information of the current frame image according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment takes how to obtain the first characteristic information of the current frame image as an example for exemplary description.
[0138] In this embodiment, the first feature information may include the current image word vector of the current frame image, and the second feature information may include the environment image word vector of the environment image.
[0139] On this basis, if Figure 2 As shown, the acquisition of the first feature information of the current frame image in the above step S101 may include the following steps S301-S302:
[0140] In step S201, feature extraction is performed on the current frame image to obtain feature points of the current frame image.
[0141] In this embodiment, after the current frame image is acquired, local features may be extracted from the image to obtain feature points of the current frame image.
[0142] In one embodiment, the specific method for extracting the above-mentioned feature points can refer to the image feature point extraction method in the related art, and this embodiment does not limit this.
[0143] In step S202, the feature points are input into a pre-trained bag-of-words model to obtain the current image word vector.
[0144] In this embodiment, after the feature points of the current frame image are obtained, the feature points can be input into a pre-trained bag-of-words model to obtain the current image word vector.
[0145] The bag-of-words model may be pre-trained based on sample data, and its input information may be feature points of an image, or the input information may be a word vector of the image.
[0146] On this basis, after obtaining the current image word vector, the first feature information described in the above step S102 is compared with the second feature information of each frame of the environmental image in the pre-constructed visual positioning map of the current environment to obtain a comparison result, which may include: determining the vector similarity between the current image word vector and the environmental image word vector.
[0147] In some embodiments, the method for determining the vector similarity between the current image word vector and the environment image word vector may include calculating the L2 distance (Euclidean distance) between the current image word vector and the environment image word vector, that is, the smaller the L2 distance, the higher the vector similarity. Exemplarily, the method for calculating the Euclidean distance between vector x and vector y can be shown in the following formula (2-1):
[0148]
[0149] In this way, the problem of comparing the first feature information of the current frame image with the second feature information of each frame of the environmental image in the visual positioning map can be transformed into the problem of comparing the current image word vector with the environmental image word vector of each frame of the environmental image in the visual positioning map, which can improve the accuracy and processing efficiency of image feature comparison, and thus improve the overall efficiency of the visual relocalization solution.
[0150] From the above description, it can be seen that this embodiment obtains the feature points of the current frame image by performing feature extraction on the current frame image, and inputs the feature points into a pre-trained bag-of-words model to obtain the current image word vector, thereby accurately extracting the current image word vector of the current frame image, and then subsequently using the current image word vector as the first feature information of the current frame image to compare with the second feature information of each frame of the environmental image in the visual positioning map, thereby obtaining the vector similarity between the current image word vector and the environmental image word vector, and then selecting a suitable image from each frame of the environmental image in the visual positioning map based on the vector similarity to relocate the mobile device, thereby avoiding the drawbacks of the relocation scheme in the related art and thus improving the relocation accuracy.
[0151] Figure 3 It is a flowchart showing how to relocate the mobile device based on the comparison result according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses how to relocate the mobile device based on the comparison result as an example to exemplify.
[0152] like Figure 3 As shown, the repositioning of the mobile device based on the comparison result in the above step S103 may include the following steps S301-S302:
[0153] In step S301, at least one candidate image is selected from each frame of the environment image based on the vector similarity.
[0154] In this embodiment, after obtaining the current image word vector, the current image word vector can be compared with the environmental image word vector of each frame of the environmental image in the visual positioning map of the current environment to obtain the vector similarity between the two, and then at least one frame of candidate image can be selected from the frames of the environmental image based on the vector similarity.
[0155] For example, after obtaining the vector similarity between the current image word vector and the environmental image word vector of each frame of the environmental image, the first s frames of candidate images with the highest similarity (i.e., the at least one frame of candidate images mentioned above) can be selected. It is worth noting that the specific value of the above s can be set based on actual needs, such as being set to 5, etc., and this embodiment does not limit this.
[0156] In step S302, the mobile device is repositioned based on the at least one candidate image frame.
[0157] In this embodiment, after at least one candidate image is selected from each frame of the environment image based on the vector similarity, the mobile device may be relocated based on the at least one candidate image.
[0158] In one embodiment, after at least one candidate image is selected from each frame of the environmental image based on the vector similarity, the posture of the current frame image relative to the visual positioning map can be determined based on the at least one candidate image, and then the mobile device can be repositioned based on the posture.
[0159] In another embodiment, the above-mentioned method of relocating the mobile device based on the at least one candidate image frame can also refer to the following Figure 4 The illustrated embodiment will not be described in detail here.
[0160] From the above description, it can be seen that this embodiment selects at least one frame of candidate image from each frame of environmental image based on the vector similarity, and relocates the mobile device based on the at least one frame of candidate image, thereby accurately relocating the mobile device based on the vector similarity between the current image word vector and the environmental image word vector, thereby improving the accuracy of relocation.
[0161] Figure 4 It is a flowchart showing how to reposition the mobile device based on the at least one frame of candidate images according to an exemplary embodiment of the present disclosure; based on the above embodiment, this embodiment uses an example of how to reposition the mobile device based on the at least one frame of candidate images to illustrate.
[0162] like Figure 4 As shown, the repositioning of the mobile device based on the at least one candidate image frame described in the above step S302 may include the following steps S401-S404:
[0163] In step S401, a feature point matching relationship between the current frame image and each of the at least one frame candidate image is determined.
[0164] In this embodiment, after at least one candidate image is selected from each frame of the environment image based on the vector similarity, a feature point matching relationship between the current frame image and each of the at least one candidate image frame can be determined.
[0165] For example, after the at least one candidate image frame is selected, the matching relationship between the feature points of the current image frame and the feature points of each candidate image frame in the at least one candidate image frame may be determined.
[0166] In some embodiments, the feature point descriptor of the current frame image can be determined, and the feature point descriptor of the current frame image can be matched with the feature point descriptor of each frame of the candidate image, so that after the successfully matched feature point descriptor is detected, the matching relationship between the feature points corresponding to the successfully matched feature point descriptor is established. Exemplarily, assuming that the feature point of the current frame image is A, its descriptor is a, and the feature point of the candidate image is B, its descriptor is b, if the descriptor a matches the descriptor b successfully, then the matching relationship between the feature point A and the feature point B can be established. In this way, the matching relationship between multiple pairs of feature points between the current frame image and each of the candidate images in at least one frame of the candidate image can be obtained.
[0167] In step S402, the relative postures corresponding to the at least one frame of candidate images are determined based on the feature point matching relationship.
[0168] In this embodiment, after determining the feature point matching relationship between the current frame image and each of the at least one candidate image frame, the relative posture corresponding to each of the at least one candidate image frame can be determined based on the feature point matching relationship.
[0169] The relative posture is the posture of the current frame image relative to the visual positioning map.
[0170] It can be understood that since the posture of each frame of the environment image in the visual positioning map is known, after obtaining the matching relationship between multiple pairs of feature points between the current frame image and each candidate image in at least one candidate image frame, the posture of the current frame image relative to the visual positioning map can be calculated based on each candidate image in the at least one candidate image frame, that is, the "posture of the current frame image relative to the visual positioning map" corresponding to each of the at least one candidate image frame.
[0171] In step S403, the relative posture is converted based on external parameters to obtain a candidate radar frame posture.
[0172] In this embodiment, after the relative posture corresponding to each of the at least one candidate image frame is determined based on the feature point matching relationship, the relative posture can be converted based on external parameters to obtain the candidate radar frame posture.
[0173] Among them, the external parameters are relative external parameters of the laser radar sensor and the image sensor.
[0174] For example, suppose that the “pose of the current frame image relative to the visual positioning map” corresponding to the i-th (i=1, 2, ..., s)-th frame candidate image is T world_camera_i , the relative external parameter of the laser radar sensor and the image sensor is T camera_lidar, the candidate radar frame pose T corresponding to the i-th candidate image can be determined based on the following formula (4-1): world_lidar_i :
[0175] T world_lidar_i =T world_camera_i *T camera_lidar ; (4-1)
[0176] In this way, s candidate radar frame poses can be obtained.
[0177] In step S404, in response to the presence of a target radar frame posture satisfying a set condition in the candidate radar frame postures, the mobile device is repositioned based on the target radar frame posture.
[0178] In this embodiment, after the relative posture is converted based on the external reference to obtain the above-mentioned s candidate radar frame postures, it can be determined whether there is a target radar frame posture that meets the set conditions among the s candidate radar frame postures. If there is a target radar frame posture that meets the set conditions, it is a correct loop, and the mobile device can be repositioned based on the target radar frame posture.
[0179] The method for determining the target radar frame posture that meets the set conditions can be referred to as follows: Figure 5 The illustrated embodiment will not be described in detail here.
[0180] It can be seen from the above description that this embodiment determines the feature point matching relationship between the current frame image and each candidate image in the at least one candidate image frame, and based on the feature point matching relationship, determines the relative posture corresponding to each of the at least one candidate image frame, and then converts the relative posture based on external parameters to obtain the candidate radar frame posture, and then responds to the presence of a target radar frame posture that meets the set conditions in the candidate radar frame posture, repositions the mobile device based on the target radar frame posture, thereby achieving repositioning of the mobile device based on the at least one candidate image frame, and improving the accuracy of repositioning.
[0181] Figure 5 is a flowchart showing how to determine whether there is a target radar frame posture that meets the set conditions according to an exemplary embodiment of the present disclosure;
[0182] Based on the above embodiment, this embodiment uses how to determine whether there is a target radar frame posture that meets the set conditions as an example to exemplify.
[0183] like Figure 5 As shown, the visual relocalization method of this embodiment may further include determining whether there is a target radar frame posture that meets the set conditions based on the following steps S501-S502:
[0184] In step S501, the candidate radar frame posture is used as the posture at the previous moment, and the current frame image is matched with the map.
[0185] In this embodiment, after obtaining the above s candidate radar frame postures, each candidate radar frame posture can be used as the posture of the previous moment (ie, the prior posture Pose0) to match the current frame image with the map.
[0186] For example, s candidate radar frame poses can be added to the pre-built laser slam system respectively, and then the current frame image is matched with the environment image in the visual positioning map.
[0187] In step S502, in response to the current frame image successfully matching the map, it is determined that the candidate radar frame posture meets the set condition, and the candidate radar frame posture is used as the target radar frame posture.
[0188] Among them, the successful matching of the above-mentioned current frame image with the map may include that the matching error between the point cloud of the current frame image and the point cloud of the environment image in the map is less than or equal to a preset error threshold, and the number of successfully matched points is greater than or equal to a set ratio.
[0189] It is worth noting that the above preset error threshold and setting ratio can be set based on actual business needs, such as setting the preset error threshold to 0.5 cm and / or setting the setting ratio to 70%, etc., which is not limited in this embodiment.
[0190] In other embodiments, when each of the candidate radar frame poses is used as the pose at the previous moment (i.e., the prior pose Pose0), after matching the current frame image with the map, if all matches fail, it can be determined that there is no target radar frame pose that meets the set conditions, that is, the above s candidate radar frame poses cannot converge, and this relocation fails.
[0191] As can be seen from the above description, this embodiment matches the current frame image with the map by taking the candidate radar frame posture as the posture at the previous moment, and in response to the successful matching of the current frame image with the map, determines that the candidate radar frame posture satisfies the set condition, and takes the candidate radar frame posture as the target radar frame posture, so as to accurately determine whether there is a target radar frame posture that satisfies the set condition in the candidate radar frame posture, and then can realize subsequent repositioning of the mobile device based on the target radar frame posture, which can improve the accuracy and precision of the repositioning.
[0192] Figure 6 is a flowchart showing how to construct the visual positioning map according to an exemplary embodiment of the present disclosure;
[0193] This embodiment is based on the above embodiment and takes how to construct the visual positioning map as an example to illustrate.
[0194] like Figure 6 As shown, the visual relocation method of this embodiment may further include constructing the visual positioning map based on the following steps S601-S603:
[0195] In step S601, the laser point cloud, the environment image sequence and the inertial data are respectively collected based on the laser radar sensor, the image sensor and the IMU inertial sensor.
[0196] In this embodiment, when constructing the visual positioning map, the mobile device can enter the current environment in advance to collect the laser point cloud, environmental image sequence and inertial data of the current environment based on the carried lidar sensor, image sensor and IMU inertial sensor.
[0197] The above environment image sequence includes multiple frames of environment images, and each frame of the environment image may correspond to a posture of the mobile device.
[0198] In step S602, based on the laser point cloud and the inertial data, a laser synchronous positioning and mapping algorithm is used to determine the device posture of the mobile device.
[0199] In this embodiment, after the laser point cloud, the environment image sequence and the inertial data are collected, the device posture of the mobile device can be determined based on the laser point cloud and the inertial data using a laser synchronous positioning and mapping algorithm.
[0200] For example, after collecting the above-mentioned laser point cloud, environmental image sequence and inertial data, the device posture of the mobile device can be determined based on the laser point cloud and the inertial data using a laser simultaneous localization and mapping (SLAM) algorithm.
[0201] In some embodiments, the laser SLAM algorithm can use the collected inertial data and combine it with the inter-frame matching of the laser point cloud data to estimate the posture of the device, so as to obtain the 6-DOF posture of the mobile device.
[0202] In step S603, a visual positioning map of the current environment is constructed based on the device posture, the laser point cloud and the environment image sequence.
[0203] In this embodiment, after the device posture of the mobile device is determined using a laser synchronous positioning and mapping algorithm based on the laser point cloud and the inertial data, a visual positioning map of the current environment can be constructed based on the device posture, the laser point cloud and the environmental image sequence.
[0204] After determining the device posture of the mobile device, the device posture, the laser point cloud and the environmental image sequence can be used to quickly build a visual positioning map, which can omit the step of three-dimensional reconstruction based on images in related technologies, and by combining the image information provided by the environmental image sequence, the error probability of laser point cloud matching can be effectively reduced. In addition, the laser point cloud can be used to directly, quickly and stably build a visual positioning map, thereby effectively avoiding the problems of long reconstruction time of the visual positioning map and easy errors in large scene mapping.
[0205] In some embodiments, the above-mentioned method of constructing the visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence can refer to the following Figure 7 The illustrated embodiment will not be described in detail here.
[0206] From the above description, it can be seen that this embodiment collects the laser point cloud, the environmental image sequence and the inertial data based on the laser radar sensor, the image sensor and the IMU inertial sensor respectively, and based on the laser point cloud and the inertial data, uses a laser synchronous positioning and mapping algorithm to determine the device posture of the mobile device, and then constructs a visual positioning map of the current environment based on the device posture, the laser point cloud and the environmental image sequence. It can realize the construction of a visual positioning map by combining laser point cloud data and image information. Compared with the repositioning scheme based only on images in the related art, it can avoid problems such as visual map skewness, ghosting, errors and long mapping time, thereby improving the accuracy of constructing a visual positioning map. On this basis, the accuracy of subsequent repositioning based on the visual positioning map can be improved.
[0207] Figure 7 is a flowchart showing how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence according to an exemplary embodiment of the present disclosure;
[0208] Based on the above embodiments, this embodiment uses an example of how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence to exemplify the above embodiments.
[0209] like Figure 7As shown, the step S603 described above of constructing the visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence may include the following steps S701-S704:
[0210] In step S701, the laser point cloud is projected onto a corresponding environment image in the environment image sequence based on the intrinsic parameters and extrinsic parameters of the image sensor to obtain a projection point.
[0211] In this embodiment, when constructing the visual positioning map of the current environment based on the device posture, the laser point cloud and the environmental image sequence, the laser point cloud can be first projected onto the corresponding environmental image in the environmental image sequence based on the intrinsic and extrinsic parameters of the image sensor to obtain the projection points. The extrinsic parameters are the relative extrinsic parameters of the laser radar sensor and the image sensor.
[0212] For example, assuming that the relative external parameters of the laser radar sensor and the image sensor are T camera_lidar , the intrinsic parameter of the image sensor is K cam , the 3D laser point cloud in the laser radar coordinate system at the current moment (e.g., moment t0) can be projected onto the environment image based on the following formula (7-1) to obtain the 2D projection point P2c:
[0213] P2c=K cam *P 3c=K cam *T camera_lidar *P lidar ; (7-1)
[0214] Among them, P3c represents the coordinates of the 3D laser point in the camera coordinate system, P2 c Represents the coordinates of the 2D point in the environment image, P lidar Represents the coordinates of the 3D laser point in the laser radar coordinate system.
[0215] In step S702, an environmental image word vector of each frame of the environmental image in the environmental image sequence is determined based on the projection point.
[0216] In this embodiment, when the laser point cloud is projected onto the corresponding environmental image in the environmental image sequence based on the intrinsic and extrinsic parameters of the image sensor and the projection points are obtained, the environmental image word vector of each frame of the environmental image in the environmental image sequence can be determined based on the projection points.
[0217] In some embodiments, after the laser point cloud is projected onto the corresponding environmental image in the environmental image sequence to obtain the projection point, the environmental image word vector of each frame of the environmental image can be determined based on the method of determining the word vector of the image in the related art. For example, the environmental image word vector of each frame of the environmental image can be determined using the coordinates of the projection point on the image and the pixel information of other points around the projection point.
[0218] In another embodiment, the above method of determining the environment image word vector of each frame of the environment image in the environment image sequence based on the projection point can also refer to the following Fig. 9 The illustrated embodiment will not be described in detail here.
[0219] In step S703, the visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
[0220] In this embodiment, after the environmental image word vector of each frame of the environmental image sequence is determined based on the projection point, the visual positioning map can be formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence. The map can be used to reposition the mobile device by comparing it with the current image frame collected by the mobile device. The specific repositioning process can be referred to above. Figure 1 The illustrated embodiments are not described in detail here.
[0221] From the above description, it can be seen that in this embodiment, the laser point cloud is projected onto the corresponding environmental image in the environmental image sequence based on the internal and external parameters of the image sensor to obtain the projection point, and the environmental image word vector of each frame of the environmental image in the environmental image sequence is determined based on the projection point, and then the visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence. It can be achieved that the visual positioning map of the current environment is accurately constructed based on the device posture, the laser point cloud and the environmental image sequence, and then the foundation can be laid for the subsequent repositioning of the mobile device based on the map, which can improve the accuracy of repositioning.
[0222] Figure 8 is a flowchart showing how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence according to another exemplary embodiment of the present disclosure;
[0223] Based on the above embodiments, this embodiment uses an example of how to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence to exemplify the above embodiments.
[0224] like Figure 8 As shown, the step S603 described above of constructing the visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence may include the following steps S801-S804:
[0225] In step S801, voxel filtering is performed on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud.
[0226] In this embodiment, when constructing a visual positioning map of the current environment based on the device posture, the laser point cloud and the environmental image sequence, considering that the excessive amount of laser point cloud data collected by the lidar sensor will lead to low efficiency of the repositioning algorithm, this embodiment performs voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud.
[0227] It is worth noting that the voxel grid size g for the voxel filtering can be set based on actual business needs, such as being set to g=1*1*1, etc., and this embodiment does not limit this.
[0228] It is understandable that voxel filtering can remove a certain degree of noise points and outliers, and its main function is to perform downsampling. Specifically, the principle is to first calculate a cube that can just wrap the point cloud based on the input point cloud, and then divide the large cube into different small cubes according to the voxel grid size g (this small cube is the "voxel grid size g"). For each point in the small cube, calculate its center of mass, and use the coordinates of the center of mass to approximately replace the coordinates of several points in the cube.
[0229] In step S802, the filtered point cloud is projected onto a corresponding environment image in the environment image sequence based on the intrinsic parameters and extrinsic parameters of the image sensor to obtain projection points.
[0230] In this embodiment, when the laser point cloud is voxel filtered based on a preset voxel grid size to obtain a filtered point cloud, the operation of projecting the laser point cloud onto the corresponding environmental image in the environmental image sequence to obtain a projection point can be performed based on the filtered point cloud instead of the laser point cloud, that is, the filtered point cloud is projected onto the corresponding environmental image in the environmental image sequence based on the intrinsic and extrinsic parameters of the image sensor to obtain a projection point. The extrinsic parameters are relative extrinsic parameters of the laser radar sensor and the image sensor.
[0231] In step S803, an environmental image word vector of each frame of the environmental image in the environmental image sequence is determined based on the projection point.
[0232] In step S804, the visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
[0233] The relevant explanations and instructions of steps S803-S804 can be found in the above Figure 7 Steps S702 - S703 in the illustrated embodiment are not described in detail here.
[0234] From the above description, it can be seen that this embodiment performs voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud, and replaces the laser point cloud based on the filtered point cloud to perform the operation of projecting the laser point cloud onto the corresponding environmental image in the environmental image sequence to obtain the projection point. This can solve the problem of low algorithm efficiency caused by excessive amount of laser point cloud data collected by the lidar sensor, and can greatly improve the efficiency of the repositioning algorithm.
[0235] Fig. 9 is a flowchart showing how to project the laser point cloud onto the corresponding environment image in the environment image sequence to obtain projection points based on the intrinsic parameters and extrinsic parameters of the image sensor according to an exemplary embodiment of the present disclosure;
[0236] Based on the above embodiments, this embodiment uses an example of how to project the laser point cloud onto the corresponding environmental image in the environmental image sequence to obtain projection points based on the internal and external parameters of the image sensor.
[0237] like Fig. 9 As shown, the step S802 described above of projecting the laser point cloud onto the corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain the projection point may include the following steps S901-S905:
[0238] In step S901, each frame of the environment image in the environment image sequence is divided into a plurality of grids based on a preset grid size;
[0239] In step S902, based on the intrinsic parameters and extrinsic parameters of the image sensor, the current laser point in the laser point cloud is projected onto a corresponding grid among a plurality of grids of the corresponding environment image to form the latest projection point, and the current number of projection points on the corresponding grid is counted;
[0240] In step S903, whether the current quantity is less than or equal to the preset quantity threshold: if so, execute step S904; if not, execute step S905;
[0241] In step S904, retaining the latest projection point;
[0242] In step S905, the latest projection point is deleted.
[0243] In this embodiment, when the laser point cloud is projected onto the corresponding environmental image in the environmental image sequence to obtain projection points, each frame of the environmental image in the environmental image sequence can be divided into multiple grids based on a preset grid size, for example, divided into m*n grids; then, the laser points in the laser point cloud can be projected one by one onto the corresponding grids in the multiple grids of the corresponding environmental image based on the intrinsic and extrinsic parameters of the image sensor to form the latest projection points, and the current number of current projection points can be counted after each projection; and then, in response to the current number being less than or equal to a preset number threshold, the latest projection point can be retained, and in response to detecting that the current number is greater than the preset number threshold, the latest projection point can be deleted, so that the number of projection points in each grid can be controlled below the preset number threshold, thereby increasing the sparsity of 2D points and improving the efficiency of the repositioning algorithm.
[0244] It can be seen from the above description that this embodiment divides each frame of the environmental image in the environmental image sequence into multiple grids based on a preset grid size, and projects the current laser point in the laser point cloud to a corresponding grid in the multiple grids of the corresponding environmental image based on the internal and external parameters of the image sensor to form the latest projection point, and counts the current number of projection points on the corresponding grid, and then when it is detected that the current number is less than or equal to the preset number threshold, retains the latest projection point, and when it is detected that the current number is greater than the preset number threshold, deletes the latest projection point, which can control the number of projection points in each grid, increase the sparsity of 2D points, and thus improve the efficiency of the repositioning algorithm.
[0245] Fig.10 is a flowchart showing how to determine the environmental image word vector of each frame of the environmental image in the environmental image sequence based on the projection point according to an exemplary embodiment of the present disclosure;
[0246] Based on the above embodiments, this embodiment uses an example of how to determine the environmental image word vector of each frame of the environmental image in the environmental image sequence based on the projection point to exemplify.
[0247] like Fig.10As shown, the step S702 described in the above may include the following steps S111-S112:
[0248] In step S111, key point descriptors at each projection position of each frame of the environment image in the environment image sequence are determined based on the projection points.
[0249] In this embodiment, when the environmental image word vector of each frame of the environmental image in the environmental image sequence is determined based on the projection point, the key point descriptor at each projection position of each frame of the environmental image in the environmental image sequence can be determined based on the projection point.
[0250] The key point descriptor can be used to encode and describe the neighborhood image block of the corresponding key point (i.e., projection point) in the environment image. The specific encoding method can be set based on actual business needs, such as directly using neighborhood pixel values for encoding, or using the HOG histogram of neighborhood pixel values, or using binary encoding, etc. This embodiment does not limit this.
[0251] In step S112, the environment image word vector is determined based on the key point descriptor.
[0252] In this embodiment, after determining the key point descriptors at each projection position of each frame of the environmental image in the environmental image sequence based on the projection points, the environmental image word vector (Bag-of-Words, Bow for short) can be determined based on the key point descriptors.
[0253] In some embodiments, when determining the environmental image word vector based on the key point descriptor, each key point descriptor can be regarded as a word, and the bag-of-words model (BOW) algorithm in the related technology can be combined to obtain the environmental image word vector.
[0254] In some other embodiments, the key point descriptor may be input into a pre-trained bag-of-words model to obtain the word vector of the environment image. The bag-of-words model may be pre-trained based on sample data, and its input information may be the key point descriptor of the image, and the input information may be the word vector of the image.
[0255] From the above description, it can be seen that this embodiment determines the key point descriptors at each projection position of each frame of the environmental image in the environmental image sequence based on the projection points, and determines the environmental image word vector based on the key point descriptors, so as to accurately determine the environmental image word vector of each frame of the environmental image in the environmental image sequence based on the projection points, and then subsequently form the visual positioning map based on the device posture, the laser point cloud, the projection points and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence, which can avoid problems such as map skewness, ghosting, errors and long map construction time, and thus improve the accuracy of map-based repositioning.
[0256] Fig.11 It is a block diagram of a visual relocalization device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be applied to a mobile device (such as a robot, an autonomous driving vehicle or a vehicle-mounted terminal, etc.) having a lidar sensor, an image sensor and an IMU inertial sensor.
[0257] like Fig.11 As shown, the device includes: an information acquisition module 110, a result acquisition module 120 and a relocation module 130, wherein:
[0258] The information acquisition module 110 is used for acquiring first feature information of the current frame image in response to the current frame image being acquired by the image sensor;
[0259] A result acquisition module 120 is used to compare the first feature information with the second feature information of each frame of the environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected based on the laser radar sensor, the image sensor, and the IMU inertial sensor;
[0260] The repositioning module 130 is configured to reposition the mobile device based on the comparison result.
[0261] From the above description, it can be known that the device of this embodiment obtains the first feature information of the current frame image in response to the current frame image collected by the image sensor, and compares the first feature information with the second feature information of each frame environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on the pre-collected current environment data, and the current environment data includes the laser point cloud, environment image sequence and inertial data of the current environment collected respectively based on the laser radar sensor, the image sensor and the IMU inertial sensor, and then the mobile device is relocated based on the comparison result. Compared with the repositioning scheme based only on images in the related art, the visual positioning map is constructed based on the characteristic data. Problems such as map skewness, ghosting, errors and long map construction time can be avoided, and the accuracy of constructing the visual positioning map can be improved, thereby improving the accuracy of subsequent repositioning based on the visual positioning map. On the other hand, since the first feature information of the current frame image collected by the image sensor is compared with the second feature information of each frame of the environmental image in the visual positioning map, repositioning can be performed based on the image features of the current frame image and each frame of the environmental image in the visual positioning map. Compared with the repositioning scheme based only on laser point clouds in the related art, the mismatch problem caused by the lack of obvious features of the laser point clouds can be avoided, thereby improving the accuracy of repositioning.
[0262] Fig.12 2 is a block diagram of another visual relocalization device according to an exemplary embodiment of the present disclosure; the device of this embodiment can be applied to a mobile device (such as a robot, an autonomous vehicle, or a vehicle-mounted terminal, etc.) having a laser radar sensor, an image sensor, and an IMU inertial sensor. Among them, the information acquisition module 210, the result acquisition module 220, and the relocalization module 230 are similar to the aforementioned Fig.11 The functions of the information acquisition module 110 , the result acquisition module 120 and the relocation module 130 in the illustrated embodiment are the same and will not be described in detail herein.
[0263] In this embodiment, the first feature information may include the current image word vector of the current frame image, and the second feature information may include the environment image word vector of the environment image.
[0264] Furthermore, if Fig.12 As shown, the information acquisition module 210 may include:
[0265] A feature point acquisition unit 211 is used to extract features from the current frame image to obtain feature points of the current frame image;
[0266] A word vector acquisition unit 212, used for inputting the feature points into a pre-trained bag-of-words model to obtain the current image word vector;
[0267] On this basis, the result acquisition module 220 can also be used to determine the vector similarity between the current image word vector and the environment image word vector.
[0268] In some embodiments, the relocation module 230 may include:
[0269] An image selection unit 231, configured to select at least one candidate image from each frame of the environment image based on the vector similarity;
[0270] The repositioning unit 232 is configured to reposition the mobile device based on the at least one candidate image frame.
[0271] In some embodiments, the relocation unit 232 may also be used to:
[0272] Determine a feature point matching relationship between the current frame image and each candidate image in the at least one candidate image frame;
[0273] Based on the feature point matching relationship, determine the relative posture corresponding to each of the at least one frame of candidate images, where the relative posture is the posture of the current frame image relative to the visual positioning map;
[0274] Converting the relative posture based on an external parameter to obtain a candidate radar frame posture, wherein the external parameter is a relative external parameter between the laser radar sensor and the image sensor;
[0275] In response to the existence of a target radar frame posture satisfying a set condition in the candidate radar frame postures, the mobile device is repositioned based on the target radar frame posture.
[0276] In some embodiments, the relocation unit 232 may also be used to:
[0277] Taking the candidate radar frame posture as the posture at the previous moment, matching the current frame image with the map;
[0278] In response to the current frame image successfully matching the map, determining that the candidate radar frame posture meets the set condition, and using the candidate radar frame posture as the target radar frame posture;
[0279] The current frame image successfully matches the map, including:
[0280] The matching error between the point cloud of the current frame image and the point cloud of the environment image in the map is less than or equal to a preset error threshold, and the number of successfully matched points is greater than or equal to a set ratio.
[0281] In some embodiments, the above apparatus may further include a map construction module 240;
[0282] The map construction module 240 may include:
[0283] A data acquisition unit 241 is used to respectively acquire the laser point cloud, the environment image sequence and the inertial data based on the laser radar sensor, the image sensor and the IMU inertial sensor;
[0284] A posture determination unit 242, configured to determine the device posture of the mobile device using a laser synchronous positioning and mapping algorithm based on the laser point cloud and the inertial data;
[0285] The map construction unit 243 is used to construct a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence.
[0286] In some embodiments, the map construction unit 243 may also be used to:
[0287] Projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point, wherein the external parameter is a relative external parameter of the laser radar sensor and the image sensor;
[0288] Determine an environment image word vector for each frame of the environment image in the environment image sequence based on the projection point;
[0289] The visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
[0290] In some embodiments, the map construction unit 243 may also be used to:
[0291] Performing voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud;
[0292] The laser point cloud is replaced based on the filtered point cloud, and the operation of projecting the laser point cloud onto a corresponding environmental image in the environmental image sequence to obtain a projection point is performed.
[0293] In some embodiments, the map construction unit 243 may also be used to:
[0294] Dividing each frame of the environmental image in the environmental image sequence into a plurality of grids based on a preset grid size;
[0295] Projecting a current laser point in the laser point cloud onto a corresponding grid among a plurality of grids of the corresponding environment image based on the internal and external parameters of the image sensor to form a latest projection point, and counting the current number of projection points on the corresponding grid;
[0296] In response to detecting that the current number is less than or equal to a preset number threshold, retaining the latest projection point.
[0297] In some embodiments, the map construction unit 243 may also be used to:
[0298] In response to detecting that the current number is greater than the preset number threshold, deleting the latest projection point.
[0299] In some embodiments, the map construction unit 243 may also be used to:
[0300] Determine, based on the projection points, key point descriptors at each projection position of each frame of the environment image in the environment image sequence;
[0301] The environment image word vector is determined based on the key point descriptor.
[0302] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0303] Fig.13 900 is a block diagram of a mobile device according to an exemplary embodiment. For example, the device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0304] Reference Fig.13 , device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .
[0305] The processing component 902 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.
[0306] The memory 904 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0307] The power supply component 906 provides power to the various components of the device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 900.
[0308] The multimedia component 908 includes a screen that provides an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display panel and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0309] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.
[0310] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0311] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 914 can detect the open / closed state of the device 900, the relative positioning of components, such as the display panel and keypad of the device 900, and the sensor assembly 914 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 914 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0312] The communication component 916 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0313] In an exemplary embodiment, the device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0314] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0315] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0316] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A visual relocalization method, It is characterized in that Applied to a mobile device having a laser radar sensor, an image sensor, and an IMU inertial sensor, the method comprises: In response to collecting a current frame image based on the image sensor, acquiring first feature information of the current frame image; Comparing the first feature information with second feature information of each frame of environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected by the laser radar sensor, the image sensor, and the IMU inertial sensor; The mobile device is relocated based on the comparison result.
2. The method according to claim 1, It is characterized in that The first feature information includes a current image word vector of the current frame image, and the second feature information includes an environment image word vector of the environment image; The obtaining of the first feature information of the current frame image includes: Performing feature extraction on the current frame image to obtain feature points of the current frame image; Inputting the feature points into a pre-trained bag-of-words model to obtain the current image word vector; The step of comparing the first feature information with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result includes: Determine the vector similarity between the current image word vector and the environment image word vector.
3. The method according to claim 2, It is characterized in that The relocating the mobile device based on the comparison result includes: Selecting at least one candidate image from each frame of the environment image based on the vector similarity; The mobile device is repositioned based on the at least one candidate image frame.
4. The method according to claim 3, It is characterized in that The repositioning of the mobile device based on the at least one candidate image frame includes: Determine a feature point matching relationship between the current frame image and each candidate image in the at least one candidate image frame; Based on the feature point matching relationship, determine the relative posture corresponding to each of the at least one frame of candidate images, where the relative posture is the posture of the current frame image relative to the visual positioning map; Converting the relative posture based on an external parameter to obtain a candidate radar frame posture, wherein the external parameter is a relative external parameter between the laser radar sensor and the image sensor; In response to the existence of a target radar frame posture satisfying a set condition in the candidate radar frame postures, the mobile device is repositioned based on the target radar frame posture.
5. The method according to claim 4, It is characterized in that The method further comprises: Taking the candidate radar frame posture as the posture at the previous moment, matching the current frame image with the map; In response to the current frame image successfully matching the map, determining that the candidate radar frame posture meets the set condition, and using the candidate radar frame posture as the target radar frame posture; The current frame image successfully matches the map, including: The matching error between the point cloud of the current frame image and the point cloud of the environment image in the map is less than or equal to a preset error threshold, and the number of successfully matched points is greater than or equal to a set ratio.
6. The method according to claim 1, It is characterized in that The method also includes pre-building the visual positioning map based on the following manner: Based on the laser radar sensor, the image sensor and the IMU inertial sensor, respectively collect the laser point cloud, the environment image sequence and the inertial data; Determine the device posture of the mobile device using a laser synchronous positioning and mapping algorithm based on the laser point cloud and the inertial data; A visual positioning map of the current environment is constructed based on the device posture, the laser point cloud and the environment image sequence.
7. The method according to claim 6, It is characterized in that The step of constructing a visual positioning map of the current environment based on the device posture, the laser point cloud and the environment image sequence includes: Projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point, wherein the external parameter is a relative external parameter of the laser radar sensor and the image sensor; Determine an environment image word vector for each frame of the environment image in the environment image sequence based on the projection point; The visual positioning map is formed based on the device posture, the laser point cloud, the projection point and the environmental image word vector corresponding to each frame of the environmental image in the environmental image sequence.
8. The method according to claim 7, It is characterized in that The method further comprises: Performing voxel filtering on the laser point cloud based on a preset voxel grid size to obtain a filtered point cloud; The laser point cloud is replaced based on the filtered point cloud, and the operation of projecting the laser point cloud onto a corresponding environmental image in the environmental image sequence to obtain a projection point is performed.
9. The method according to claim 7, It is characterized in that The step of projecting the laser point cloud onto a corresponding environment image in the environment image sequence based on the internal and external parameters of the image sensor to obtain a projection point includes: Dividing each frame of the environmental image in the environmental image sequence into a plurality of grids based on a preset grid size; Projecting a current laser point in the laser point cloud onto a corresponding grid among a plurality of grids of the corresponding environment image based on the internal and external parameters of the image sensor to form a latest projection point, and counting the current number of projection points on the corresponding grid; In response to detecting that the current number is less than or equal to a preset number threshold, retaining the latest projection point.
10. The method according to claim 9, It is characterized in that The method further comprises: In response to detecting that the current number is greater than the preset number threshold, deleting the latest projection point.
11. The method according to claim 7, It is characterized in that The step of determining the environmental image word vector of each frame of the environmental image in the environmental image sequence based on the projection point includes: Determine, based on the projection points, key point descriptors at each projection position of each frame of the environment image in the environment image sequence; The environment image word vector is determined based on the key point descriptor.
12. A visual repositioning device, It is characterized in that Applied to a mobile device having a laser radar sensor, an image sensor, and an IMU inertial sensor, the device comprises: An information acquisition module, configured to acquire first feature information of a current frame image in response to the current frame image being acquired by the image sensor; A result acquisition module is used to compare the first feature information with the second feature information of each frame of the environment image in the pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on the pre-collected current environment data, and the current environment data includes the laser point cloud, the environment image sequence and the inertial data of the current environment respectively collected based on the laser radar sensor, the image sensor and the IMU inertial sensor; A repositioning module is used to reposition the mobile device based on the comparison result.
13. A mobile device, It is characterized in that The device comprises: LiDAR sensor, image sensor, IMU inertial sensor, processor and memory for storing computer programs; Wherein, the processor is configured to implement, when executing the computer program: In response to collecting a current frame image based on the image sensor, acquiring first feature information of the current frame image; Comparing the first feature information with second feature information of each frame of environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected by the laser radar sensor, the image sensor, and the IMU inertial sensor; The mobile device is relocated based on the comparison result.
14. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by the processor, it realizes: In response to collecting a current frame image based on an image sensor, acquiring first feature information of the current frame image; Comparing the first feature information with second feature information of each frame of the environment image in a pre-constructed visual positioning map of the current environment to obtain a comparison result, wherein the visual positioning map is constructed based on pre-collected current environment data, and the current environment data includes a laser point cloud, an environment image sequence, and inertial data of the current environment respectively collected based on a laser radar sensor, the image sensor, and an IMU inertial sensor; The mobile device is relocated based on the comparison result.
Citation Information
Patent Citations
Vehicle repositioning method and device
CN108416808A
Robot rapid repositioning method and system based on visual dictionary
CN110533722A
Self-adaptive reinforced fusion real-time instance segmentation method based on camera and laser radar
CN111967373A
Pose determination method and device, electronic equipment and storage medium
CN111983635A
Semantic map construction method based on laser and vision fusion
CN115187737A
