Relocation method, device and equipment for cross-modal feature fusion

Abstract

The invention relates to a cross-modal feature fusion relocation method, device and equipment, and the method comprises the steps: carrying out the preprocessing of perception data of a data sensor of mobile equipment, and obtaining point cloud data fused with depth information; performing feature fusion with the two-dimensional visual feature descriptor of each point cloud according to a geometric relationship between the point cloud data to obtain geometric-visual joint features corresponding to the geometric figure; the geometric figure is composed of a plurality of point clouds; in a key frame database constructed in real time or a global point cloud map database constructed in advance, comprehensive matching of geometric-visual joint features is carried out; and performing pose calculation according to a target graph pair formed by the target geometric graph and the target matching graph which are successfully and comprehensively matched to obtain calibration pose information of the mobile equipment. The method can be applied to loopback detection in a visual SLAM system, map multiplexing and repositioning can be carried out based on the generated three-dimensional point cloud map, high-precision positioning is achieved, and the method has high illumination robustness.

Description

Technical Field

[0001] The present disclosure relates to the field of positioning and mapping technology, and in particular to a repositioning method, apparatus, and device for cross-modal feature fusion. Background Art

[0002] Simultaneous localization and mapping (SLAM) is a key technology in fields such as autonomous driving, intelligent robotics, drones, and virtual reality. It uses sensors to perceive the surrounding environment and estimate pose. However, because visual sensors and laser sensors are external relative sensors, pose estimation suffers from recursive and cumulative errors. Over long periods of operation, the estimated state of the system can deviate from the true value. Therefore, pose estimation requires correction.

[0003] In the process of realizing the concept of the present disclosure, the inventors found that there are at least the following technical problems in the related art: in the related art, the system often corrects the deviated posture through loop detection or map reuse repositioning of a single sensor; the visual perception-based method needs to combine historical three-dimensional landmark points to solve the posture, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints can easily cause drift errors; the lidar perception method is not robust enough in scenes with repetitive structures (such as walls and corridors), and the mismatch rate is high; in some cases, ignoring the dynamic changes in sensor reliability (such as sparse lidar point clouds or camera image occlusion, etc.) can easily lead to matching failure problems. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, embodiments of the present disclosure provide a cross-modal feature fusion relocalization method, apparatus, and device.

[0005] In the first aspect, an embodiment of the present disclosure provides a repositioning method for cross-modal feature fusion. The above method includes: pre-processing the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information; performing feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the above geometric figure is composed of multiple point clouds; performing comprehensive matching of the above geometric-visual joint features in the key frame database constructed in real time or the pre-constructed global point cloud map database; performing pose solution on the target figure composed of the target geometric figure that successfully matches the target matching figure to obtain the calibration pose information of the above mobile device.

[0006] In some embodiments, the point cloud data is feature point cloud data fused with depth information from the perspective of the camera imaging plane of the mobile device; the feature point cloud data is generated from at least one of visual perception data and radar perception data. The method comprises performing feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to the geometric figure, including: performing position matching and association between the feature point cloud data fused with depth information and the two-dimensional visual feature descriptor extracted from the image pixels from the perspective of the camera imaging plane; and selecting a point cloud based on the results of the position matching and association to construct a geometric figure for feature fusion to obtain a geometric-visual joint feature corresponding to the geometric figure.

[0007] In some embodiments, based on the results of position matching and association, point clouds are selected to construct geometric figures for feature fusion to obtain geometric-visual joint features corresponding to the geometric figures, including:

[0008] For a first point cloud object whose position matches the feature point cloud data fused with the depth information and the corresponding position of the two-dimensional visual feature descriptor, the first point cloud object is selected to construct a geometric figure;

[0009] For a second point cloud object in which the position of the feature point cloud data fused with the depth information does not match the position corresponding to the two-dimensional visual feature descriptor, performing positional nearest neighbor association based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and constructing a geometric figure based on the result of the positional nearest neighbor association;

[0010] The constructed geometric figure is feature spliced according to at least the following dimensions: point cloud position, edge length, and two-dimensional visual feature descriptors corresponding to the contained point clouds to obtain geometric-visual joint features corresponding to the geometric figure.

[0011] In some embodiments, performing positional neighbor association based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and constructing a geometric figure based on the result of the positional neighbor association includes:

[0012] When the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object are within a preset neighboring range, the point clouds in the first target point cloud and the second target point cloud that are within the preset neighboring range are marked as first associated point clouds, and the multidimensional features of the first associated point clouds are shared within the preset neighboring range; the first associated point clouds are selected within the preset neighboring range to construct a geometric figure; or,

[0013] When there is a positional overlap between the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object, expansion is performed within a preset neighborhood based on the positional overlap in the first target point cloud and the second target point cloud, and the point cloud within the preset neighborhood is marked as a second associated point cloud, and the multidimensional features of the above-mentioned second associated point cloud are shared within the preset neighborhood; the second associated point cloud is selected in the above-mentioned preset neighborhood to construct a geometric figure.

[0014] In some embodiments, the above-mentioned selection of point clouds to construct geometric figures includes: arbitrarily selecting a preset number of point cloud data to form a set of candidate geometric figures; sorting the edges between point clouds in each candidate geometric figure in the above-mentioned set according to the size of the edges; according to the sorting of the edges, determining in sequence whether any edge in each candidate geometric figure exceeds a set threshold; eliminating candidate geometric figures with edges exceeding the set threshold, and constructing the remaining candidate geometric figures in the above-mentioned set as geometric figures to be matched.

[0015] In some embodiments, the comprehensive matching includes: a first matching for the geometric relationship and a second matching for the two-dimensional visual feature descriptor, and the first matching and the second matching are performed progressively according to a preset order.

[0016] In some embodiments, the above-mentioned geometric figure is a triangle formed by randomly selecting three point clouds; the geometric-visual joint features corresponding to the above-mentioned geometric figure include: the positions of the three point cloud vertices in the triangle, the lengths of the three connecting edges and the two-dimensional visual feature descriptors corresponding to the three point cloud vertices; in the above-mentioned key frame database or the above-mentioned global point cloud map database, the geometric-visual joint features of each reference triangle are stored based on the storage hash index obtained by processing the diagonal value of the longest side. Among them, in the key frame database constructed in real time or the global point cloud map database constructed in advance, the above-mentioned geometric-visual joint features are comprehensively matched, including: for each triangle to be matched, the following comprehensive matching processing operations are performed: the diagonal value of the longest side of the current triangle to be matched is processed to obtain the hash index to be matched; the above-mentioned hash index to be matched is matched with the stored hash index and located in the target storage interval; within the above-mentioned target storage interval, angle matching is performed based on the diagonal value of the longest side of the current triangle to be matched; for the first matching reference triangle set obtained by successful angle matching, side length matching is performed in sequence based on the longest side and the second longest side of the current triangle to be matched; for the second matching reference triangle set obtained by successful side length matching, feature matching is performed based on the two-dimensional visual feature descriptor of each point cloud vertex of the triangle to be matched; based on the target matching reference triangle obtained by successful feature matching and the current triangle, a target graphic pair is generated.

[0017] In some embodiments, a pose calculation is performed on a target graphic pair consisting of a target geometric graphic and a target matching graphic that have been successfully matched, to obtain the calibration pose information of the above-mentioned mobile device, including: calling a PNP algorithm to perform pose calculation based on the correspondence between the target geometric graphic and the target matching graphic in the target graphic pair, to obtain the calibration pose information of the above-mentioned mobile device at the target moment; or, in the case where there are multiple groups of target graphic pairs, calling a RANSAC algorithm to eliminate mismatched pairs, and performing pose optimization solution on the screened matching pairs based on a nonlinear optimization algorithm to obtain the calibration pose information of the above-mentioned mobile device at the target moment.

[0018] In some embodiments, the perception data includes at least one of the following: visual perception data and radar perception data; wherein the visual perception data includes at least one of the following: image data captured by a depth camera and image data captured by a multi-camera. The preprocessing of the perception data of the data sensor to obtain point cloud data fused with depth information includes at least one of the following: performing pixel feature point extraction and feature description on the image data captured by the depth camera, and projecting the pixels into three-dimensional space according to the corresponding depth values to obtain feature point cloud data fused with depth information; or performing pixel feature point extraction and feature description on the image data captured by the multi-camera, and performing depth generation and projection into three-dimensional space based on the corresponding stereo vision of the multi-camera to obtain feature point cloud data fused with depth information; or using the point cloud data in the radar perception data as a three-dimensional feature point cloud, projecting the three-dimensional feature point cloud back to the imaging plane corresponding to the camera and performing feature description to obtain feature point cloud data fused with depth information.

[0019] In a second aspect, an embodiment of the present disclosure provides a repositioning device with cross-modal feature fusion. The above-mentioned repositioning device is integrated on a mobile device or is an independent and communicative body with the mobile device. The above-mentioned repositioning device includes: a point cloud data generation module, a feature fusion module, a comprehensive matching module and a pose solution module. The above-mentioned point cloud data generation module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information. The above-mentioned feature fusion module is used to perform feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the above-mentioned geometric figure is composed of multiple point clouds. The above-mentioned comprehensive matching module is used to perform comprehensive matching of the above-mentioned geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database. The above-mentioned pose solution module is used to perform pose solution based on the target figure pair composed of the target geometric figure that has been successfully comprehensively matched and the target matching figure to obtain the calibrated pose information of the above-mentioned mobile device.

[0020] In a third aspect, an embodiment of the present disclosure provides a mobile device. The mobile device is integrated with the repositioning device provided by the embodiment of the second aspect; or, the mobile device includes: a first point cloud data generation module, a first feature fusion module, a first comprehensive matching module, and a first pose solving module. The first point cloud data generation module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information. The first feature fusion module is used to perform feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the geometric figure is composed of multiple point clouds. The first comprehensive matching module is used to perform comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database. The first pose solving module is used to perform pose solving based on the target figure pair composed of the target geometric figure that has been successfully comprehensively matched and the target matching figure to obtain the calibrated pose information of the mobile device.

[0021] In some embodiments, the mobile device is at least one of the following: an intelligent robot, a vehicle, a flying device, or a wearable device.

[0022] In a fourth aspect, embodiments of the present disclosure provide a service device. The service device is configured to provide service support for positioning of a mobile device, and the service device integrates the repositioning device provided by the embodiment of the second aspect. Alternatively, the service device includes: a second point cloud data generation module, a second feature fusion module, a second comprehensive matching module, and a second pose calculation module. The second point cloud data generation module is configured to preprocess the sensor data of the mobile device to obtain point cloud data fused with depth information. The second feature fusion module is configured to perform feature fusion based on the geometric relationships between the point cloud data and the two-dimensional visual feature descriptors of each point cloud to obtain geometric-visual joint features corresponding to a geometric figure composed of multiple point clouds. The second comprehensive matching module is configured to perform comprehensive matching of the geometric-visual joint features within a real-time constructed keyframe database or a pre-constructed global point cloud map database. The second pose calculation module is configured to perform pose calculation based on a target figure pair consisting of a successfully comprehensively matched target geometric figure and a target matching figure, to obtain calibrated pose information for the mobile device.

[0023] In a fifth aspect, embodiments of the present disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory is configured to store a computer program; and the processor is configured to implement the cross-modal feature fusion relocalization method described above when executing the program stored in the memory.

[0024] In a sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-modal feature fusion relocalization method described above.

[0025] The above technical solutions provided by the embodiments of the present disclosure have at least some or all of the following advantages:

[0026] By utilizing the geometric relationship between the point cloud data fused with depth information and the two-dimensional visual feature descriptor of each point cloud for feature fusion, a multi-dimensional fusion of three-dimensional geometric features and two-dimensional visual feature descriptors is achieved. The multi-dimensional feature description of the point cloud is performed synchronously based on the three-dimensional geometric features and two-dimensional visual feature descriptors that incorporate depth information. In this way, when subsequently matching with the keyframe database or the global point cloud map database for position recognition, matching based on the geometric-visual joint features corresponding to the geometric figures can make up for the defect that similar scenes such as white walls and corridors cannot be distinguished when matching based solely on geometric features. At the same time, since the matching of the two-dimensional visual feature descriptor also combines the depth information corresponding to the three-dimensional geometric features, the redundancy of a large number of point cloud positions is reduced during matching, the matching efficiency and matching accuracy are improved, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints is easily caused by drift errors. Overall, the processing efficiency and accuracy of real-time positioning and pose estimation can be improved. This method can not only be applied to loop detection in visual SLAM systems, but can also be used for map reuse and relocalization based on the already generated three-dimensional point cloud map (supporting color point cloud maps), achieving high-precision positioning with strong illumination robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0028] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 The flowchart of the cross-modal feature fusion relocalization method according to an embodiment of the present disclosure is schematically shown.

[0030] Figure 2 The detailed implementation flow chart of step S120 according to an embodiment of the present disclosure is schematically shown.

[0031] Figure 3 The detailed implementation flow chart of step S220 according to an embodiment of the present disclosure is schematically shown.

[0032] Figure 4 Schematic diagram showing a comparison of the position recognition and loop closure detection results obtained by (a) processing the relocalization method provided in an embodiment of the present invention for a radar-vision dataset collected in reality and (b) obtaining the position recognition and loop closure detection results obtained by simply using the STD triangle descriptor.

[0033] Figure 5 The structure block diagram of the cross-modal feature fusion relocalization device according to an embodiment of the present disclosure is schematically shown.

[0034] Figure 6 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0036] During research and development, it was found that in related technologies, positioning systems often correct deviated postures through loop closure detection of a single sensor or map reuse repositioning; visual perception-based methods need to combine historical three-dimensional landmark points to solve the posture, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints can easily cause drift errors; the lidar perception method is not robust enough in scenes with repetitive structures (such as walls and corridors) and has a high mismatch rate; in some cases, ignoring the dynamic changes in sensor reliability (such as sparse lidar point clouds or occlusion of camera images, etc.) can easily lead to matching failure problems.

[0037] For example, some lidar perception methods rely on the geometric features of laser point cloud data and do not consider visual information. In some scenarios, relying solely on geometric features may not effectively distinguish locations with similar geometric structures, such as corridors and walls, and is prone to high mismatch rates. Furthermore, when applied to vision systems, this method lacks the ability to directly locate objects in a 3D point cloud map based on visual observations. The vision system cannot directly use a simple 3D point cloud to associate with the current image, making pose optimization (or relocalization) impossible.

[0038] For example, some purely visual perception methods use ORB features extracted and a bag-of-words model for encoding. This approach relies on the quality of bag-of-words training: building a bag-of-words model requires a large amount of training data, and the quality of this training directly impacts the accuracy of loop closure detection. If the training data differs significantly from the actual application scenario, detection performance may degrade.

[0039] In the pure visual perception method, in the process of position recognition and matching based on ORB features, due to the very large amount of point cloud data, higher-dimensional features are generally extracted, such as using bag-of-words models, semantic recognition, etc., and the binary descriptors corresponding to the position points will not be used alone. Otherwise, the amount of calculation will be very large and the retrieval efficiency will be very low. Generally, existing visual systems do not support this method. In other words, in the pure visual perception method, due to the extremely large amount of point cloud data, it is basically necessary to extract the point cloud features in high dimensions before using them. The point cloud data will not be used directly, otherwise the overall system will crash and the positioning and matching operations will not be possible. Moreover, the pure visual perception solution cannot be directly reused for three-dimensional point cloud maps. For example, maps generated by laser point clouds cannot be reused, and the applicable scenarios are limited.

[0040] In addition, since most visual perception methods rely on two-dimensional image features for position recognition, the lack of depth information makes it difficult to be directly compatible with the three-dimensional point cloud map constructed by lidar. It is necessary to use relevant algorithms and combine historical three-dimensional landmark points to solve the position and posture. The scale uncertainty of pure 2D-2D geometric constraints can easily cause drift errors.

[0041] In view of this, the embodiments of the present disclosure provide a repositioning method, device and equipment for cross-modal feature fusion, which fuses three-dimensional geometric features with two-dimensional visual feature descriptors in multiple dimensions. It can not only be applied to loop detection in visual SLAM systems, but also can perform map reuse and repositioning based on the already generated three-dimensional point cloud map (supporting color point cloud maps) to achieve high-precision positioning. The method provided by the embodiments of the present disclosure integrates the respective advantages of three-dimensional geometric features and two-dimensional visual features and complements each other by fusing three-dimensional geometric features with two-dimensional visual feature descriptors, improves the processing efficiency and accuracy of real-time positioning and pose estimation, has strong lighting robustness, supports the reuse of pure geometric point cloud maps (such as maps generated by lidar), and direct matching of color point cloud maps (such as color point cloud maps generated by systems such as depth cameras (RGB-D), LV-SLAM (lidar-visual-inertial odometry) systems).

[0042] By synchronously performing multi-dimensional feature description of the point cloud based on three-dimensional geometric features and two-dimensional visual feature descriptors that incorporate depth information, when subsequently matching with a keyframe database or a global point cloud map database for position recognition, matching based on the geometric-visual joint features corresponding to the geometric figures can make up for the defect that similar scenes such as white walls and corridors cannot be distinguished when matching based solely on geometric features; at the same time, since the matching of two-dimensional visual feature descriptors will also combine the depth information corresponding to the three-dimensional geometric features, the redundancy of a large number of point cloud positions is reduced during matching, the matching efficiency and matching accuracy are improved, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints that are prone to drift errors is avoided; overall, the processing efficiency and accuracy of real-time positioning and pose estimation can be improved. This method can not only be applied to loop detection in visual SLAM systems, but can also be used for map reuse and repositioning based on the already generated three-dimensional point cloud map (supporting color point cloud maps), achieving high-precision positioning with strong illumination robustness.

[0043] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0044] The first exemplary embodiment of the present disclosure provides a relocalization method using cross-modal feature fusion. The method provided in this embodiment can be applied to an electronic device with computing capabilities, which can be integrated into a mobile device or be a device that is independent of and capable of communicating with the mobile device, to provide computing services for positioning the mobile device.

[0045] Figure 1 The flowchart of the cross-modal feature fusion relocalization method according to an embodiment of the present disclosure is schematically shown.

[0046] Reference Figure 1 As shown, the cross-modal feature fusion relocalization method provided by the embodiment of the present disclosure includes the following steps: S110, S120, S130 and S140.

[0047] In step S110 , the perception data of the data sensor of the mobile device is preprocessed to obtain point cloud data fused with depth information.

[0048] In some embodiments, data sensors may include: one or more types of TOF (time-of-flight) sensors (which measure distance by measuring the time required for light, sound waves, or other types of signals to be transmitted and received), such as optical, acoustic, and radar sensors, visual sensors, radars, etc. The perception data corresponding to the mobile device may be, but is not limited to, one or more of the following: image data captured by a depth camera (RGB-D camera) (an example of visual perception data), image data captured by a multi-camera (another example of visual perception data), and radar perception data. In some embodiments, radar perception data may be a sparse point cloud output by LV-SLAM (lidar-visual-inertial odometry), or point cloud data output by one or more radars (e.g., lidar, millimeter-wave radar, etc.).

[0049] In some embodiments, the above preprocessing includes but is not limited to: point cloud generation, plane segmentation, feature extraction, feature description, etc.

[0050] In the above step S110, the image data captured by the depth camera is preprocessed on the perception data of the data sensor to obtain point cloud data fused with depth information, including:

[0051] The image data taken by the depth camera is subjected to pixel feature point extraction and feature description, and the pixels are projected into three-dimensional space according to the corresponding depth values to obtain feature point cloud data fused with depth information.

[0052] During the depth camera shooting, each pixel will have a corresponding depth value. Based on the image data taken by the depth camera, pixel feature points are extracted and described, and the depth values are combined to project into three-dimensional space. Feature point cloud data with fused depth information can be obtained from the camera imaging plane perspective of the mobile device.

[0053] In the above step S110, the perception data of the data sensor is preprocessed for the image data captured by the multi-camera to obtain point cloud data fused with depth information, including:

[0054] The pixel feature points and feature description are extracted from the image data taken by multiple cameras, and the depth is generated and projected into three-dimensional space based on the stereo vision corresponding to multiple cameras to obtain feature point cloud data that integrates depth information.

[0055] Multi-eye stereo vision composed of multiple cameras can generate pixel-corresponding depth. Based on the image data taken by multiple cameras, pixel feature points are extracted and described, and the depth values are combined and projected into three-dimensional space. Feature point cloud data with fused depth information can be obtained from the camera imaging plane perspective of the mobile device.

[0056] In the above step S110, the radar sensing data is pre-processed on the sensing data of the data sensor to obtain point cloud data fused with depth information, including:

[0057] The point cloud data in the radar perception data is used as a three-dimensional feature point cloud, which is projected back to the imaging plane corresponding to the camera and described to obtain feature point cloud data fused with depth information.

[0058] The point cloud data in the radar perception data is the three-dimensional feature point cloud that carries depth information. It is the perception result in the world coordinate system. By projecting the three-dimensional feature point cloud back to the imaging plane corresponding to the camera, the corresponding position under the perspective of the camera imaging plane is obtained and the feature description is performed. In this way, feature point cloud data with integrated depth information can be obtained under the perspective of the camera imaging plane of the mobile device.

[0059] In step S120, feature fusion is performed based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to the geometric figure; the above-mentioned geometric figure is composed of multiple point clouds.

[0060] In some embodiments, the point cloud data is feature point cloud data fused with depth information from the perspective of the camera imaging plane of the mobile device; the feature point cloud data is generated by at least one of visual perception data and radar perception data.

[0061] The information corresponding to the above-mentioned two-dimensional visual feature descriptors can be various types of information that describe point cloud features in a two-dimensional perspective, which can be 64-dimensional, 128-dimensional, 256-dimensional, etc. CNN (convolutional neural network) features (for point clouds with color and grayscale, it can express differences), or various binary descriptors, or other improved two-dimensional feature description information, etc.

[0062] Since the two-dimensional visual feature descriptor extracted from the image pixels under the camera imaging plane perspective is represented as another dimensional feature of the point cloud, geometric figures are constructed and feature fusion is performed after position matching and association with the previously obtained feature point cloud data, thereby realizing multi-dimensional fusion of three-dimensional geometric features and two-dimensional visual feature descriptors and subsequent comprehensive matching.

[0063] A geometric shape is formed by selecting several point clouds from the feature point cloud data. By varying the number of selected point clouds, different geometric shapes can be generated. These geometric shapes can include, but are not limited to, triangles, quadrilaterals, pentagons, hexagons, and other polygons, as well as irregular shapes. To reduce the amount of matching computation, regular shapes are preferred. By varying the positions of the selected point clouds while keeping the number of selected point clouds constant, multiple geometric shapes of the same type can be generated.

[0064] The three-dimensional geometric features are obtained by describing the geometric relationships of each geometric figure. The above-mentioned geometric relationships include but are not limited to at least one of the following: vertex positions, side lengths, angle distribution, corresponding relationships between sides and angles, etc. Considering that matching in the database based solely on three-dimensional geometric features will be affected by factors such as lighting and occlusion, in the embodiment of the present disclosure, feature fusion is performed based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figures, thereby realizing multi-dimensional fusion of three-dimensional geometric features and two-dimensional visual feature descriptors, and performing multi-dimensional feature description of the point cloud based on the three-dimensional geometric features integrated with depth information and the two-dimensional vision. In this way, when matching with the key frame database or the global point cloud map database for position recognition in the future, it is necessary to comprehensively match the multi-dimensional features.

[0065] Taking a triangle as an example of a geometric figure, the geometric-visual joint feature of a triangle is expressed as [p, l, I], where p represents the position p(p1, p2, p3) of the three point cloud vertices that constitute the triangle; l represents the length of the three sides l(l1, l2, l3) of the three point cloud vertices that constitute the triangle; and I represents the two-dimensional visual feature descriptor I(I1, I2, I3) corresponding to the three point cloud vertices that constitute the triangle.

[0066] Figure 2 The detailed implementation flow chart of step S120 according to an embodiment of the present disclosure is schematically shown.

[0067] In some embodiments, reference Figure 2 As shown, in the above step S120, feature fusion is performed based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figures, including the following steps: S210 and S220.

[0068] In step S210 , position matching and association are performed based on the feature point cloud data fused with the depth information and the two-dimensional visual feature descriptors extracted from the image pixels under the camera imaging plane viewing angle.

[0069] In some embodiments, the following situations may exist between the position corresponding to the two-dimensional visual feature descriptor extracted for the image pixel under the camera imaging plane perspective (also described as the point cloud position) and the point cloud position corresponding to the feature point cloud data fused with depth information: some point cloud positions match and other point cloud positions do not match.

[0070] The position matching process is to match the corresponding point cloud positions of two modalities. One modality is the position corresponding to the feature point cloud data obtained based on the fusion of depth information, and the other modality is the two-dimensional visual feature descriptor extracted from the image pixels (which can be the image pixels in the image obtained based on the visual perception data) under the perspective of the camera imaging plane; the corresponding point cloud positions are matched based on the two modalities, and the subsequent step S311 is executed for the matched part to construct a geometric figure; the position association processing is performed on the unmatched part, and the subsequent step S312 is executed to also construct a geometric figure; then step S320 is executed.

[0071] Based on the description of the above-mentioned step S110, it can be known that the above-mentioned point cloud data is the feature point cloud data fused with depth information under the perspective of the camera imaging plane of the above-mentioned mobile device. Therefore, the above-mentioned geometric relationship is also the relationship between the edges and angles of the feature point cloud data fused with depth information under the perspective of the camera imaging plane. By integrating the two-dimensional visual feature descriptor under another modality into the edge and angle relationship of the feature point cloud data fused with depth information and performing position matching and association, geometric figures are constructed for feature fusion. It is possible to comprehensively reflect the point cloud features in the imaging plane from a more comprehensive and mutually complementary multi-dimensional perspective. On the one hand, since the geometric-visual joint feature fuses the three-dimensional geometric features with depth information and the features that can represent color and light, the feature cloud features in the imaging plane can be comprehensively reflected ... It supports the reuse of pure geometric point cloud maps (such as maps generated by lidar) and direct matching of color point cloud maps (such as color point cloud maps generated by depth cameras (RGB-D), LV-SLAM (lidar-visual-inertial odometry) systems, etc.); when encountering sparse lidar point clouds, due to the comprehensive matching of geometric-visual joint features, it can assist and accurately realize position recognition and matching based on the modality corresponding to the two-dimensional visual feature descriptor; when encountering camera image occlusion, feature fusion can realize the mutual complementation of two-dimensional vision and geometry, making the features more comprehensive and reducing the probability of mismatching during comprehensive matching.

[0072] In step S220, point clouds are selected based on the results of position matching and association to construct geometric figures for feature fusion, thereby obtaining geometric-visual joint features corresponding to the geometric figures.

[0073] Figure 3 The detailed implementation flow chart of step S220 according to an embodiment of the present disclosure is schematically shown.

[0074] In some embodiments, reference Figure 3As shown, in the above step S220, point clouds are selected to construct geometric figures based on the results of position matching and association for feature fusion, and geometric-visual joint features corresponding to the geometric figures are obtained, including the following steps: S311, S312, and S320, wherein steps S311 and S312 are parallel steps and can be executed one by one; in some scenarios, there will be situations where both S311 and S312 are included.

[0075] In step S311 , for a first point cloud object whose position matches the feature point cloud data fused with depth information and the corresponding position of the two-dimensional visual feature descriptor, the first point cloud object is selected to construct a geometric figure.

[0076] For example, the feature point cloud data fused with depth information is described as a set M, and the point cloud position corresponding to the two-dimensional visual feature descriptor extracted for the image pixels under the perspective of the camera imaging plane is described as N. There is a point cloud position set Mc of partial fused depth information and a point cloud position set Nc corresponding to the two-dimensional visual feature descriptor. The positions of the two are completely overlapped, and the positions of the two are regarded as matching. Then Mc and Nc can be regarded as the first point cloud object, or each point cloud in Mc and Nc can be regarded as the first point cloud object, and point clouds are selected from the set DY1 composed of the first point cloud objects to construct geometric figures.

[0077] In step S312, for the second point cloud object whose feature point cloud data fused with depth information does not match the position corresponding to the two-dimensional visual feature descriptor, position neighbor association is performed based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and a geometric figure is constructed based on the result of the position neighbor association.

[0078] Continuing with the above example, there is also a situation where the positions of the point cloud position set Md that is partially fused with depth information do not completely overlap with the positions in the point cloud position set Nd corresponding to the two-dimensional visual feature descriptor. The two are considered to have mismatched positions. Then Md and Nd can be regarded as the second point cloud object, or each point cloud in Md and Nd can be regarded as the second point cloud object.

[0079] In some embodiments, in step S312, performing positional neighbor association based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and constructing a geometric figure based on the result of the positional neighbor association includes:

[0080] When the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object are within a preset neighboring range, the point clouds in the first target point cloud and the second target point cloud that are within the preset neighboring range are marked as first associated point clouds, and the multidimensional features of the first associated point clouds are shared within the preset neighboring range; the first associated point clouds are selected within the preset neighboring range to construct a geometric figure; or,

[0081] When there is a positional overlap between the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object, expansion is performed within a preset neighborhood based on the positional overlap in the first target point cloud and the second target point cloud, and the point cloud within the preset neighborhood is marked as a second associated point cloud, and the multidimensional features of the above-mentioned second associated point cloud are shared within the preset neighborhood; the second associated point cloud is selected in the above-mentioned preset neighborhood to construct a geometric figure.

[0082] In the above embodiment, there are some feature point cloud data in the second point cloud object whose first position distribution and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor are in a neighbor relationship or a neighbor relationship with partial position overlap. By marking the point cloud within the preset neighbor range as an associated point cloud (corresponding to the first associated point cloud or the second associated point cloud), and sharing multiple features of the associated point cloud within the neighbor range, it is possible to effectively avoid the situation where single-modal feature data points are missing in some scenarios, and reduce the problem of mismatching based on multi-modal feature neighbor sharing and comprehensive matching.

[0083] In some embodiments, in step S220, selecting a point cloud to construct a geometric figure includes:

[0084] A set of candidate geometric figures is formed by randomly selecting a preset number of point cloud data; the preset number can be 3, 4, 5, 6, etc., to form candidate geometric figures such as triangles, quadrilaterals, pentagons, and hexagons. By selecting point clouds at different positions and combining them, a set containing multiple candidate geometric figures is obtained. In this set, some candidate geometric figures may share edges or points. In fact, in the random generation method, there are many cases where candidate geometric figures share edges or points.

[0085] In the above set, the edges between the point clouds in each candidate geometric figure are sorted according to the size of the edges;

[0086] According to the order of the connected edges, determine in order whether any connected edge in each candidate geometric figure exceeds the set threshold;

[0087] The candidate geometric figures whose edges exceed the set threshold are eliminated, and the remaining candidate geometric figures in the above set are constructed as the geometric figures to be matched.

[0088] In this embodiment, based on a set threshold, the candidate geometric figures in the above set are screened according to the edge connection size, and the candidate geometric figures with any edge connection exceeding the set threshold are eliminated. The remaining candidate geometric figures in the above set are determined as the geometric figures to be matched, which can reduce the number of geometric figures to be matched and improve the matching efficiency. At the same time, since the geometric figures corresponding to the overly long side lengths can actually be decomposed into geometric figures with smaller side lengths during matching, in the case where there are a large number of shared edges and shared points among the candidate geometric figures in the above set, data redundancy can be reduced by eliminating the geometric figures with overly long side lengths, and only matching is performed based on geometric figures with finer granularity, effectively improving the matching efficiency and reducing data redundancy.

[0089] In step S320, the constructed geometric figures are feature - stitched at least according to the following dimensions: point cloud position, edge connection side length, and the two - dimensional visual feature descriptor corresponding to the contained point cloud, to obtain the geometric - visual joint feature corresponding to the geometric figure.

[0090] Taking a triangle as an example of a geometric figure, p represents the positions p(p1, p2, p3) of the three point cloud vertices forming the triangle, and each position can be expressed by three - dimensional coordinates x, y, z; l represents the three side lengths l(l1, l2, l3) of the three point cloud vertices forming the triangle. In some embodiments, each triangle is arranged in ascending order of side length. For example, there is the following sorting relationship: l1 ≤ l2 < l3; I represents the two - dimensional visual feature descriptors I(I1, I2, I3) corresponding to the three point cloud vertices forming the triangle. The position of each point cloud vertex and the two - dimensional visual feature descriptor are corresponding one by one in the order of side length, where the point corresponds to the opposite side; then the geometric - visual joint feature of the triangle is represented as [p, l, I].

[0091] In step S130, in the key - frame database constructed in real - time or the pre - constructed global point cloud map database, the above - mentioned comprehensive matching of geometric - visual joint features is performed.

[0092] In some application scenarios of the present disclosure, for in - process analysis, when performing simultaneous localization and mapping processing on a mobile device during movement, steps S110 - S140 are executed. The perception data in step S110 is real - time perception data, that is, the time stamp corresponding to each perception data is the current moment; in step S130, it can be a key - frame database constructed in real - time or a pre - constructed global point cloud map database (for example, in an offline mode, the pre - constructed global point cloud map database is imported, and during the real - time movement of the mobile device, mapping is no longer performed and only the pose is determined). This method can correspond to the loop - closure detection mode: constructing a key - frame database in real - time, storing the geometric - visual joint features of geometric figures and the corresponding pose T.

[0093] In other application scenarios, post-analysis is performed after the mobile device completes a task or mobile mapping. The analysis focuses on the mobile device's movements. The perception data in step S110 is historical perception data, and the timestamps corresponding to each perception data point correspond to historical moments and are fixed. Step S130 can utilize a pre-built global point cloud map database. This approach corresponds to a map reuse relocalization mode: a global point cloud map database is pre-generated to store the geometric-visual joint features and corresponding poses T of the geometric figures.

[0094] The above-mentioned databases are described as: i |P i =(l i ,I i ,p i ,T i ),i∈n},P i Represents the matching reference information stored in the database; i represents the frame number; n is the total number of frames; T i Indicates the pose corresponding to the i-th frame. For the matching reference information in the database (described as matching pair information), it can be distinguished from the joint features of the point cloud to be matched by adding a subscript target.

[0095] In some embodiments, the comprehensive matching includes: a first matching for the geometric relationship and a second matching for the two-dimensional visual feature descriptor, and the first matching and the second matching are performed progressively according to a preset order.

[0096] In some embodiments, the above-mentioned geometric figure is a triangle formed by randomly selecting three point clouds; the geometric-visual joint features corresponding to the above-mentioned geometric figure include: the positions of the three point cloud vertices in the triangle, the lengths of the three connecting edges and the two-dimensional visual feature descriptors corresponding to the three point cloud vertices; in the above-mentioned key frame database or the above-mentioned global point cloud map database, the storage hash index obtained based on the diagonal value of the longest side is used to store the geometric-visual joint features and corresponding poses (which can also be described as matching pair information) of each reference triangle.

[0097] In the above step S130, the comprehensive matching of the above-mentioned geometric-visual joint features is performed in the key frame database constructed in real time or the pre-constructed global point cloud map database, including:

[0098] For each triangle to be matched, perform the following comprehensive matching processing operations:

[0099] Process the diagonal value of the longest side of the current triangle to be matched to obtain the hash index to be matched;

[0100] Match the above-mentioned hash index to be matched with the storage hash index to locate the target storage interval;

[0101] In the target storage interval, angle matching is performed based on the diagonal value of the longest side of the current triangle to be matched;

[0102] For the first matching reference triangle set obtained by successful angle matching, perform side length matching in sequence based on the longest side and the second longest side of the current triangle to be matched;

[0103] For the second matching reference triangle set obtained by successful side length matching, feature matching is performed based on the two-dimensional visual feature descriptors of each point cloud vertex of the triangle to be matched;

[0104] The target matching reference triangle and the current triangle obtained based on the successful feature matching are used to generate a target graphic pair.

[0105] In this embodiment, in order to improve the overall computing efficiency and query efficiency, the cosine value of the diagonal value θ of the longest side is stored and calculated. The cosine value satisfies the following expression:

[0106]

[0107] Among them, taking a triangle as an example of a geometric figure, the positions of the three point cloud vertices are p1, p2, and p3 respectively, the opposite side of the point cloud vertex p1 is l1, the opposite side of the point cloud vertex p2 is l2, and the opposite side of the point cloud vertex p3 is l3; the diagonal value θ of the longest side falls in the angle interval (0, 180°), and cos(θ) is in the monotonically decreasing interval of (-1, 1).

[0108] The above diagonal cosine value is quantized to 0.01 (the specific value is as an example and can be changed) to form a storage hash index, and 201 intervals (including boundaries) can be obtained. Each interval corresponds to a storage unit. Some of the preset angle ranges can be described as storage intervals, and the matching pair information containing the same hash index is stored in the corresponding storage interval.

[0109] When querying, first calculate the hash value for the cosine value corresponding to the maximum angle and perform a matching query in the storage interval. After locating the target storage interval, perform a detailed angle query: perform angle matching based on the diagonal value of the longest side of the current triangle to be matched; if the current query cosine angle cos(θ q )(q represents the number of the current triangle) and the target reference cosine angle cos(θ target ) is less than the angle matching threshold thr, and the specific expression is as follows:

[0110] cos(θ q )-cos(θ target ) <thr, (2)

[0111] Matching obtains the first matching reference triangle set, in which the longest side of the current triangle to be matched is obtained. and the second longest side Match the edge lengths in sequence. The specific expressions are as follows:

[0112]

[0113] If the longest side Match reference information with a longest edge The result of the match is less than the longest edge matching threshold Then continue with the second longest side The side length matching is as follows:

[0114]

[0115] If the second longest side Match reference information with a second longest edge The result of the matching is less than the second longest edge matching threshold Get the second matching reference triangle set; continue to perform two-dimensional visual feature descriptor I on the set q The specific expression is as follows:

[0116]

[0117] If the two-dimensional visual feature descriptor I corresponding to the i-th vertex among the three vertices of the current triangle q-i and the two-dimensional visual feature descriptor I of the corresponding vertices among the three vertices of a reference triangle target-i The sum of the squared differences between the two values according to the number of vertices is less than the feature matching threshold r thr , the result of feature matching is considered to be a successful match, and the target graphic pair corresponding to the target triangle and the target matching reference triangle can be obtained.

[0118] Although the above examples use triangles as geometric figures, the solutions of the embodiments of the present disclosure can be applied to geometric figures of other shapes. For triangles, hash processing based on angles can be performed to divide multiple storage intervals and index the partitions to improve overall computing efficiency and query efficiency, which has a positive effect on processing time in real-time mapping.

[0119] In step S140, a pose calculation is performed on a target figure pair consisting of a target geometric figure that has been successfully matched and a target matching figure to obtain calibration pose information of the mobile device.

[0120] In some embodiments, in step S140, performing pose calculation on a target figure pair formed by a successfully matched target geometric figure and a target matching figure to obtain calibration pose information of the mobile device includes:

[0121] Calling the PNP algorithm to perform pose calculation based on the correspondence between the target geometric figure and the target matching figure in the target figure pair to obtain the calibration pose information of the mobile device at the target time; or,

[0122] When there are multiple sets of target graphic pairs, the RANSAC algorithm is called to eliminate mismatched pairs, and the posture optimization solution of the screened matching pairs is performed based on a nonlinear optimization algorithm (such as the Levenberg-Marquardt algorithm) to obtain the calibrated posture information of the above-mentioned mobile device at the target time.

[0123] In the embodiment including steps S110 to S140, feature fusion is performed by utilizing the geometric relationship between the point cloud data fused with depth information and the two-dimensional visual feature descriptor of each point cloud to achieve multi-dimensional fusion of three-dimensional geometric features and two-dimensional visual feature descriptors, and multi-dimensional feature description of the point cloud is performed synchronously based on the three-dimensional geometric features and two-dimensional visual feature descriptors that incorporate depth information. In this way, when subsequently matching with the key frame database or the global point cloud map database for position recognition, matching based on the geometric-visual joint features corresponding to the geometric figures can make up for the shortcomings of matching based solely on geometric features for similar scenes such as white walls and corridors. At the same time, since the matching of two-dimensional visual feature descriptors also combines the depth information corresponding to three-dimensional geometric features, the redundancy of a large number of point cloud positions is reduced during matching, the matching efficiency and accuracy are improved, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints is avoided, which easily leads to drift errors. Overall, it can improve the processing efficiency and accuracy of real-time positioning and pose estimation. This method can not only be applied to loop detection in visual SLAM systems, but also can be used for map reuse and re-positioning based on the generated three-dimensional point cloud map (supporting color point cloud maps), achieving high-precision positioning with strong illumination robustness. Position recognition and loop detection based on vision-lidar can still be performed in scenes with similar structures. At the same time, the map established based on this joint feature is also compatible with conventional position recognition schemes such as STD (geometric method) and visual observation, achieving more efficient map utilization.

[0124] Figure 4 Schematic diagram showing a comparison of the position recognition and loop closure detection results obtained by (a) processing the relocalization method provided in an embodiment of the present invention for a radar-vision dataset collected in reality and (b) obtaining the position recognition and loop closure detection results obtained by simply using the STD triangle descriptor.

[0125] contrast Figure 4 As can be seen from (a) and (b), when repositioning is performed based on the radar-vision dataset collected by the front-end LIVO system, and simply using the STD triangle descriptor (corresponding to the geometric method) for position recognition and posture correction, the number of position recognitions is large and there are a large number of errors, resulting in poor and divergent positioning results. In comparison, the method provided in the embodiment of the present disclosure is used for position recognition and loop detection, which can improve the accuracy of position recognition, achieve high-precision positioning, and have strong illumination robustness.

[0126] A second exemplary embodiment of the present disclosure provides a cross-modal feature fusion relocalization device.

[0127] Figure 5 The structure block diagram of the cross-modal feature fusion relocalization device according to an embodiment of the present disclosure is schematically shown.

[0128] Reference Figure 5 As shown, the cross-modal feature fusion relocalization device 500 provided by the embodiment of the present disclosure includes: a point cloud data generation module 510, a feature fusion module 520, a comprehensive matching module 530 and a pose solution module 540.

[0129] The relocation device is integrated into the mobile device or is an independent entity that can communicate with the mobile device.

[0130] The point cloud data generation module 510 is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information.

[0131] The feature fusion module 520 is used to perform feature fusion based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the geometric figure is composed of multiple point clouds.

[0132] The comprehensive matching module 530 is used to perform comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database.

[0133] The above-mentioned posture solving module 540 is used to perform posture solving based on the target figure pair consisting of the target geometric figure that has been successfully matched and the target matching figure, so as to obtain the calibration posture information of the above-mentioned mobile device.

[0134] For more details of this embodiment, please refer to the relevant description of the first embodiment, which will not be repeated here.

[0135] In the above-mentioned device, multi-dimensional feature description of the point cloud is performed synchronously based on the three-dimensional geometric features and two-dimensional visual feature descriptors that incorporate depth information. In this way, when subsequently matching with the key frame database or the global point cloud map database for position recognition, matching is performed based on the geometric-visual joint features corresponding to the geometric figures, which can make up for the defect that similar scenes such as white walls and corridors cannot be distinguished when matching based solely on geometric features; at the same time, since the matching of the two-dimensional visual feature descriptor will also be combined with the depth information corresponding to the three-dimensional geometric features, the redundancy of a large number of point cloud positions is reduced during matching, the matching efficiency and matching accuracy are improved, and the scale uncertainty of pure 2D (two-dimensional)-2D geometric constraints is avoided. It is easy to cause drift errors; overall, it can improve the processing efficiency and accuracy of real-time positioning and pose estimation. This method can not only be applied to loop detection in visual SLAM systems, but can also be used for map reuse and repositioning based on the already generated three-dimensional point cloud map (supporting color point cloud maps), to achieve high-precision positioning with strong illumination robustness.

[0136] A third exemplary embodiment of the present disclosure provides a mobile device. The mobile device is integrated with the repositioning device provided in the second exemplary embodiment; or the mobile device includes: a first point cloud data generation module, a first feature fusion module, a first comprehensive matching module, and a first pose solving module.

[0137] The first point cloud data generation module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information.

[0138] The first feature fusion module is used to perform feature fusion based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the geometric figure is composed of multiple point clouds.

[0139] The first comprehensive matching module is used to perform comprehensive matching of the above-mentioned geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database.

[0140] The first pose solving module is used to solve the pose of the target graphic pair composed of the target geometric graphic and the target matching graphic that have been successfully matched, so as to obtain the calibration pose information of the mobile device.

[0141] In some embodiments, the above-mentioned mobile device is at least one of the following: an intelligent robot (for example, a search and rescue robot, a transport robot, a sweeping robot, etc.), a vehicle (for example, a vehicle that supports automatic driving or assisted driving), a flying device (for example, a drone, a manned aircraft, etc.), and a wearable device.

[0142] For more details of this embodiment, please refer to the relevant description of the first embodiment, which will not be repeated here.

[0143] A fourth exemplary embodiment of the present disclosure provides a service device.

[0144] The above-mentioned service device is used to provide service support for the positioning of mobile devices, and the above-mentioned service device integrates the repositioning device provided by the above-mentioned second embodiment; or, the above-mentioned service device includes: a second point cloud data generation module, a second feature fusion module, a second comprehensive matching module and a second posture solution module.

[0145] The second point cloud data generating module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information.

[0146] The second feature fusion module is used to perform feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figure; the geometric figure is composed of multiple point clouds.

[0147] The second comprehensive matching module is used to perform comprehensive matching of the above-mentioned geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database.

[0148] The second posture solving module is used to solve the posture of the target graphic pair composed of the target geometric graphic and the target matching graphic that have been successfully matched, so as to obtain the calibration posture information of the mobile device.

[0149] For more details of this embodiment, please refer to the relevant description of the first embodiment, which will not be repeated here.

[0150] Any multiple can be merged in a module and realize in the functional module that above-mentioned relocation device, mobile device, service equipment comprise, perhaps any one module wherein can be split into multiple modules.Perhaps, at least part of the function of one or more modules in these modules can be combined with at least part of the function of other modules, and realize in a module.At least one in the functional module that above-mentioned relocation device, mobile device, service equipment comprise can be realized as hardware circuit at least in part, for example field programmable gate array (FPGA), programmable logic array (PLA), system on chip, system on substrate, system on encapsulation, application specific integrated circuit (ASIC), or can realize by hardware or firmware such as any other reasonable manner that circuit is integrated or encapsulated, or realize with any one or with wherein several appropriate combinations in software, hardware and firmware three kinds of implementations.Perhaps, at least one in the functional module that above-mentioned relocation device, mobile device, service equipment comprise can be realized as computer program module at least in part, when this computer program module is run, can perform corresponding function.

[0151] A fifth exemplary embodiment of the present disclosure provides an electronic device.

[0152] Figure 6 The structural block diagram of the electronic device provided by the embodiment of the present disclosure is schematically shown.

[0153] Reference Figure 6 As shown, the electronic device 600 provided by the embodiment of the present disclosure includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602 and the memory 603 communicate with each other through the communication bus 604; the memory 603 is used to store computer programs; the processor 601 is used to implement the cross-modal feature fusion relocation method as described above when executing the program stored in the memory.

[0154] A sixth exemplary embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned cross-modal feature fusion relocalization method.

[0155] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments, or may exist independently without being incorporated into the device or apparatus. The computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0156] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided by the embodiments of this disclosure all comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken with respect to user personal information to prevent unauthorized access to user personal information data and to maintain the security of user personal information, network security, and national security.

[0158] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0159] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not to be limited to the embodiments shown herein, but is to be construed in the broadest manner consistent with the principles and novel features claimed herein.

Claims

1. A cross-modal feature fusion relocalization method, characterized in that: include: Preprocess the sensor data of the mobile device to obtain point cloud data fused with depth information; Performing feature fusion based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to a geometric figure; the geometric figure is composed of multiple point clouds; Performing comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database; The pose of the target graphic pair formed by the successfully matched target geometric graphic and the target matching graphic is solved to obtain the calibration pose information of the mobile device.

2. The relocation method according to claim 1, wherein: The point cloud data is feature point cloud data fused with depth information under the perspective of the camera imaging plane of the mobile device; The feature point cloud data is generated by at least one of visual perception data and radar perception data; Among them, feature fusion is performed based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain the geometric-visual joint features corresponding to the geometric figures, including: Position matching and association are performed based on the feature point cloud data fused with depth information and the two-dimensional visual feature descriptors extracted from the image pixels under the camera imaging plane perspective; According to the results of position matching and association, point clouds are selected to construct geometric figures for feature fusion, and the geometric-visual joint features corresponding to the geometric figures are obtained.

3. The relocation method according to claim 2, wherein: Based on the results of position matching and association, point clouds are selected to construct geometric figures for feature fusion, and the corresponding geometric-visual joint features are obtained, including: For a first point cloud object whose position matches the feature point cloud data fused with the depth information and the corresponding position of the two-dimensional visual feature descriptor, the first point cloud object is selected to construct a geometric figure; For a second point cloud object in which the position of the feature point cloud data fused with the depth information does not match the position corresponding to the two-dimensional visual feature descriptor, performing positional nearest neighbor association based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and constructing a geometric figure based on the result of the positional nearest neighbor association; The constructed geometric figure is feature spliced according to at least the following dimensions: point cloud position, edge length, and two-dimensional visual feature descriptors corresponding to the contained point clouds to obtain geometric-visual joint features corresponding to the geometric figure.

4. The relocation method according to claim 3, wherein: Performing positional neighbor association based on the first position distribution of the feature point cloud data and the second position distribution of the point cloud corresponding to the two-dimensional visual feature descriptor, and constructing a geometric figure based on the result of the positional neighbor association, including: When the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object are within a preset neighboring range, the point clouds in the first target point cloud and the second target point cloud that are within the preset neighboring range are marked as first associated point clouds, and the multidimensional features of the first associated point clouds are shared within the preset neighboring range; the first associated point clouds are selected within the preset neighboring range to construct a geometric figure; or When there is a positional overlap between the first position distribution of the first target point cloud and the second position distribution of the second target point cloud in the second point cloud object, expansion is performed within a preset neighborhood based on the positional overlap in the first target point cloud and the second target point cloud, and the point cloud within the preset neighborhood is marked as a second associated point cloud, and the multidimensional features of the second associated point cloud are shared within the preset neighborhood; the second associated point cloud is selected in the preset neighborhood to construct a geometric figure.

5. The relocation method according to any one of claims 2 to 4, characterized in that: The step of selecting a point cloud to construct a geometric figure includes: A preset number of point cloud data are randomly selected to form a set of candidate geometric figures; sorting the edges between the point clouds in each candidate geometric figure in the set according to the size of the edges; According to the order of the connected edges, determine in order whether any connected edge in each candidate geometric figure exceeds the set threshold; The candidate geometric figures with edges exceeding a set threshold are eliminated, and the remaining candidate geometric figures in the set are constructed as geometric figures to be matched.

6. The relocation method according to claim 1, wherein: The comprehensive matching includes: a first matching for the geometric relationship and a second matching for the two-dimensional visual feature descriptor, and the first matching and the second matching are performed progressively according to a preset order.

7. The relocation method according to any one of claims 1 to 4, characterized in that: The geometric figure is a triangle formed by randomly selecting three point clouds; the geometric-visual joint features corresponding to the geometric figure include: the positions of the three point cloud vertices in the triangle, the lengths of the three connecting edges, and the two-dimensional visual feature descriptors corresponding to the three point cloud vertices; in the key frame database or the global point cloud map database, the geometric-visual joint features of each reference triangle are stored based on a storage hash index obtained by processing the diagonal value of the longest side; The comprehensive matching of the geometric-visual joint features is performed in the key frame database constructed in real time or the pre-constructed global point cloud map database, including: For each triangle to be matched, perform the following comprehensive matching processing operations: Process the diagonal value of the longest side of the current triangle to be matched to obtain the hash index to be matched; Match the to-be-matched hash index with the stored hash index to locate the target storage interval; Performing angle matching within the target storage interval based on the diagonal value of the longest side of the current triangle to be matched; For the first matching reference triangle set obtained by successful angle matching, perform side length matching in sequence based on the longest side and the second longest side of the current triangle to be matched; For the second matching reference triangle set obtained by successful side length matching, feature matching is performed based on the two-dimensional visual feature descriptors of each point cloud vertex of the triangle to be matched; The target matching reference triangle and the current triangle obtained based on the successful feature matching are used to generate a target graphic pair.

8. The relocation method according to any one of claims 1 to 4, characterized in that: Performing pose calculation on a target figure pair consisting of a target geometric figure that has been successfully matched and a target matching figure to obtain calibration pose information of the mobile device includes: Calling the PNP algorithm to perform pose calculation based on the correspondence between the target geometric figure and the target matching figure in the target figure pair to obtain the calibration pose information of the mobile device at the target time; or When there are multiple sets of target graphic pairs, the RANSAC algorithm is called to eliminate mismatched pairs, and the posture optimization solution is performed on the screened matching pairs based on a nonlinear optimization algorithm to obtain the calibration posture information of the mobile device at the target time.

9. The relocation method according to any one of claims 1 to 4, characterized in that: The perception data includes at least one of the following: visual perception data, radar perception data; wherein the visual perception data includes at least one of the following: image data taken by a depth camera, image data taken by a multi-camera; The preprocessing of the sensor's perception data to obtain point cloud data fused with depth information includes at least one of the following: Extract pixel feature points and describe features of the image data captured by the depth camera, and project the pixels into three-dimensional space according to the corresponding depth values to obtain feature point cloud data fused with depth information; or, Pixel feature point extraction and feature description are performed on the image data taken by multiple cameras, and depth is generated and projected into three-dimensional space based on the stereo vision corresponding to multiple cameras to obtain feature point cloud data fused with depth information; or, The point cloud data in the radar perception data is used as a three-dimensional feature point cloud, and the three-dimensional feature point cloud is projected back to the imaging plane corresponding to the camera and feature description is performed to obtain feature point cloud data fused with depth information.

10. A cross-modal feature fusion relocalization device, characterized in that: The relocation device is integrated into the mobile device or is an independent and communicative entity with the mobile device, and the relocation device includes: The point cloud data generation module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information; A feature fusion module is used to fuse features based on the geometric relationship between point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to a geometric figure; the geometric figure is composed of multiple point clouds; A comprehensive matching module, configured to perform comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database; The pose solving module is used to solve the pose of the target graphic pair composed of the target geometric graphic and the target matching graphic that are successfully matched, so as to obtain the calibration pose information of the mobile device.

11. A mobile device, characterized in that: The mobile device is integrated with the relocation device according to claim 10; or, The mobile device comprises: a first point cloud data generation module, a first feature fusion module, a first comprehensive matching module and a first pose solving module; The first point cloud data generation module is used to pre-process the perception data of the data sensor of the mobile device to obtain point cloud data fused with depth information; The first feature fusion module is used to perform feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to the geometric figure; the geometric figure is composed of multiple point clouds; The first comprehensive matching module is used to perform comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database; The first posture solving module is used to perform posture solving based on the target graphic pair consisting of the target geometric graphic that has been successfully matched and the target matching graphic, so as to obtain the calibration posture information of the mobile device.

12. The mobile device according to claim 11, wherein: The mobile device is at least one of the following: an intelligent robot, a vehicle, a flying device, and a wearable device.

13. A service device, characterized in that: Used to provide service support for positioning of mobile devices, the service device being integrated with the repositioning device according to claim 10; or, The service device includes: a second point cloud data generation module, a second feature fusion module, a second comprehensive matching module and a second posture solving module; The second point cloud data generating module is configured to pre-process the sensing data of the data sensor of the mobile device to obtain point cloud data fused with depth information; The second feature fusion module is used to perform feature fusion based on the geometric relationship between the point cloud data and the two-dimensional visual feature descriptor of each point cloud to obtain a geometric-visual joint feature corresponding to the geometric figure; the geometric figure is composed of multiple point clouds; The second comprehensive matching module is used to perform comprehensive matching of the geometric-visual joint features in a key frame database constructed in real time or a pre-constructed global point cloud map database; The second posture solving module is used to perform posture solving based on the target graphic pair consisting of the target geometric graphic that has been successfully matched and the target matching graphic, so as to obtain the calibration posture information of the mobile device.

14. An electronic device, characterized in that: The processor, the communication interface, the memory and the communication bus are connected to each other via the communication bus. Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 9 when executing a program stored in a memory.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images