Semantic-based loop detection method, device, electronic device and storage medium

Abstract

The present disclosure provides a semantic-based loop detection method, including: obtaining posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data; constructing a posture graph based on the posture transformation information between adjacent laser frames, optimizing the posture graph, and constructing a global map based on the optimized posture graph; and obtaining a preset number of candidate frames that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculating the similarity between the current frame and each candidate frame, and judging whether a loop occurs. The present disclosure also provides a semantic-based loop detection device, an electronic device, and a readable storage medium.

Description

Technical Field

[0001] The present disclosure relates to the fields of autonomous driving, robotics and artificial intelligence, and in particular to a semantic-based loop detection method, device, electronic device and storage medium. Background Art

[0002] SLAM (Simultaneous Localization And Mapping) technology has important applications in fields with autonomous positioning requirements, such as autonomous driving and robotics.

[0003] The core issue of SLAM is the pose estimation of mobile devices (autonomous driving vehicles, robots, etc.). Mapping is based on pose estimation and relies on sensors (radar, cameras, etc.) to perceive the environment and describe the recorded results.

[0004] Loop closure detection is mainly used to solve the problem of pose estimation drifting over time. In the process of continuous pose calculation by the front end, pose drift occurs due to sensor errors and other reasons. When the mobile device passes through the position it has passed before, there is an error between the position at this time and the previously recorded position. After this error is found, it can be sent to the back end for optimization to remove the pose drift error between two loop closure detections. Loop closure detection is a method for mobile devices to detect scenes they have been to, and is an important part of back end optimization to eliminate accumulated errors.

[0005] For example, in the prior art 1, GOSMatch: Graph-of-Semantics Matching for Detecting Loop Closures in 3D LiDAR data, a loop detection method based on semantic graph matching is published. In this method, only three semantic categories are selected, and semantic point cloud graphs are formed through semantic segmentation and clustering. The semantic point cloud graph is represented by an undirected complete graph, and the edges between each vertex represent the spatial Euclidean distance between semantic instances, and a graph descriptor based on a histogram is generated. The horizontal axis of the histogram is the vertex category combination, and the vertical axis is the statistic based on the edge length; similarly, the vertex descriptor in the graph is calculated, and the vertex descriptor is also based on the histogram, and the method is similar to the graph descriptor. Finally, the RANSAC-based method is used to calculate whether a loop occurs. This method only counts the distance information in the semantic graph, because the vehicle is also in the semantic category, and does not consider the scene differences caused by the movement of the vehicle, which may cause a decrease in the loop recall rate; at the same time, the loop detection method requires the current frame and the historical frame to use RANSAC-based calculations continuously, which is time-consuming.

[0006] Prior art 2: Chinese patent application CN112767485A discloses a method for creating a point cloud map and identifying a scene based on static semantic information, including step 1: jointly projecting a laser point cloud and a semantic image to obtain a semantic point cloud; step 2: filtering robust static information through semantic tags; step 3: extracting descriptors of robust static information and combining them into a descriptor of the current scene; step 4: combining the semantic point cloud with the odometer of the laser radar to generate a global map, and relocating on the global map through point cloud registration. This technology extracts robust static information to establish a descriptor, and the descriptor is directly matched to obtain the pose of the current frame, but the descriptor content is strongly dependent on the position of the laser radar, which can easily cause mismatching of the descriptor, and the direct matching method will consume a lot of time, which has certain limitations in actual use.

[0007] Both of the above two technical solutions cannot directly calculate the relative posture relationship between the current frame and the loop frame; and the existing technology 1 does not deal with movable obstacles such as vehicles, and the scene changes caused by movable obstacles will cause loop failure; the existing technology 2 has low calculation efficiency and is prone to mismatching.

[0008] In summary, common loop detection schemes generally have the following problems: 1) With the lidar as the center, the observation angle greatly affects the recognition rate of the loop; 2) After the loop is detected, the relative pose relationship cannot be directly calculated;

[0009] 3) The changes caused by dynamic moving objects to the scene are not considered. For scenes with vehicles, the movement of the vehicles will cause matching errors; 4) Direct matching during loop detection results in excessive computational consumption. Summary of the invention

[0010] In order to solve at least one of the above technical problems, the present disclosure provides a semantic-based loop detection method, device, electronic device and storage medium.

[0011] According to one aspect of the present disclosure, a semantic-based loop closure detection method is provided, comprising:

[0012] S110, acquiring the posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data (i.e., acquiring the posture conversion relationship between adjacent laser frames);

[0013] S120, constructing a pose graph based on at least the pose transformation information between adjacent laser frames, optimizing the pose graph, and constructing a global map based on the optimized pose graph;

[0014] S130, obtaining a preset number of candidate frames (TopN candidate frames) that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculating the similarity between the current frame and each candidate frame, and determining whether a loop occurs.

[0015] According to at least one embodiment of the present disclosure, the semantic-based loop detection method further includes: S140, if a loop occurs, obtaining a relative pose error between two frames where the loop occurs, and using the relative pose error to optimize the pose graph in step S120.

[0016] According to at least one embodiment of the semantic-based loop closure detection method of the present disclosure, S110, acquiring pose transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data, includes:

[0017] S111, performing time synchronization (preferably through timestamps) on the acquired sensor data of SLAM-related sensors including laser radar (including the multi-frame laser point cloud data described above) and performing point cloud distortion compensation on the multi-frame laser point cloud data;

[0018] S112, extracting features from each frame of laser point cloud data to obtain line and surface features of each frame of laser point cloud data;

[0019] S113, performing feature matching calculation on each frame of laser point cloud data to obtain the posture transformation information between two adjacent frames.

[0020] According to at least one embodiment of the semantic-based loop closure detection method of the present disclosure, S120, constructing a pose graph based on at least pose transformation information between adjacent laser frames, optimizing the pose graph, and constructing a global map based on the optimized pose graph, including:

[0021] S121, obtaining at least one key frame in the multiple frames of laser point cloud data (which can be obtained based on a preset key frame acquisition standard), taking the key frame pose as a node of a pose graph and optimizing the key frame pose;

[0022] S122. Projecting the laser point cloud of the key frame onto the map based on the optimized key frame pose to obtain a global map.

[0023] According to at least one embodiment of the semantics-based loop detection method of the present disclosure, S130, based on semantic information, obtains a preset number of candidate frames (TopN candidate frames) that are most similar to the current frame from the acquired multi-frame laser point cloud data, calculates the similarity between the current frame and each candidate frame, and determines whether a loop occurs, including:

[0024] S131, constructing a global descriptor of the current scene with the main object as the center;

[0025] S132, constructing a semantic category vector (semantic category vector) of the main object based on the global descriptor and the category and distance of the semantic object closest to the main object;

[0026] S133: Determine the semantic category required for semantic segmentation based on the current scene, and divide the 360-degree range around the main object into N equal parts with the main object as the center. S Equally divided regions, obtaining a semantic category histogram based on the semantic categories of the semantic objects in each of the equally divided regions and the number of semantic objects in each semantic category;

[0027] S134 (this step is to perform rough matching based on the semantic category vector), calculate the semantic category vector similarity between the semantic category vector of the main object in the current frame and the historical semantic category vector (historical semantic category vector) having the same semantic category as the main object in the acquired multi-frame laser point cloud data (i.e., historical key frames) (the same processing is performed for all main objects in the current frame), so as to obtain the historical key frames whose semantic category vector similarity is greater than or equal to the threshold similarity as candidate frames;

[0028] S135 , calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the main object in each candidate frame having the same semantic category as the main object, to obtain the historical key frame with the maximum semantic category histogram similarity.

[0029] According to at least one embodiment of the present disclosure, the semantic-based loop detection method further includes: S136, determining whether the distance between the semantic category histogram of the historical key frame with the largest semantic category histogram similarity and the semantic category histogram of the current frame is less than or equal to a threshold distance; if so, taking the historical key frame with the largest semantic category histogram similarity as a loop frame (i.e., a loop key frame).

[0030] According to at least one embodiment of the present disclosure, the semantic-based loop detection method, in step S134, further includes:

[0031] Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame; and / or,

[0032] Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame.

[0033] According to at least one embodiment of the present disclosure, the semantic-based loop detection method, in step S134, further includes:

[0034] Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the historical key frame. If so, the vehicle label is not considered when calculating the semantic category vector similarity between the current frame and the historical key frame. If not, the vehicle label is considered; and / or,

[0035] Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the historical key frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, the vehicle label is not considered when calculating the similarity of the semantic category vector between the current frame and the historical key frame. If not, the vehicle label is considered.

[0036] According to at least one embodiment of the present disclosure, the semantic-based loop detection method, S135, calculates the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the same semantic category as the main object in each candidate frame to obtain the historical key frame with the largest semantic category histogram similarity, including:

[0037] Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the candidate frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the candidate frame. If so, calculate the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category as the main object without including the vehicle-related area in the similarity calculation. If not, include the vehicle-related area in the similarity calculation; and / or,

[0038] Determine whether there is a vehicle label in the semantic category vector of the main object of the candidate frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the candidate frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, calculate the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category as the main object without including the vehicle-related area in the similarity calculation. If not, include the vehicle-related area in the similarity calculation.

[0039] According to the semantic-based loop detection method of at least one embodiment of the present disclosure, S140, if a loop occurs, obtain the relative pose error between the two frames where the loop occurs, including: for the current frame and the loop frame of the current frame, take out the pose of the main object in the current frame coordinate system and the loop frame coordinate system, and obtain the relative pose error.

[0040] According to another aspect of the present disclosure, a semantic-based loop detection device is provided, comprising:

[0041] A posture transformation information acquisition module, which acquires posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data;

[0042] A pose graph / map construction module, wherein the pose graph / map construction module constructs a pose graph based on at least pose transformation information between adjacent laser frames, optimizes the pose graph, and constructs a global map based on the optimized pose graph;

[0043] The loop detection module obtains a preset number of candidate frames that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculates the similarity between the current frame and each candidate frame, and determines whether a loop occurs.

[0044] According to at least one embodiment of the present disclosure, the semantic-based loop detection device further includes: a posture difference acquisition module, which, if a loop occurs, obtains the relative posture error between the two frames where the loop occurs.

[0045] According to at least one embodiment of the present disclosure, the semantic-based loop closure detection device further includes: a sensor device, which is at least used to acquire laser point cloud data.

[0046] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes any one of the loop detection methods described above.

[0047] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the execution instructions are used to implement any of the loop detection methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0049] Figure 1 It is a flowchart of a semantic-based loop closure detection method according to an embodiment of the present invention.

[0050] Figure 2 It is a flowchart of a semantic-based loop closure detection method according to another embodiment of the present invention.

[0051] Figure 3 It is a flowchart for obtaining posture transformation information in a semantic-based loop closure detection method according to an embodiment of the present invention.

[0052] Figure 4 This is a process for constructing a pose graph / map in a semantic-based loop closure detection method according to an embodiment of the present invention.

[0053] Figure 5 It is a flowchart of determining a loop frame in a semantic-based loop detection method according to an embodiment of the present disclosure.

[0054] Figure 6 It is a flowchart of determining a loop frame in a semantic-based loop detection method according to another embodiment of the present disclosure.

[0055] Figure 7 It is a semantic object diagram of a single main object and all the fan-shaped areas around it.

[0056] Figure 8 yes Figure 7 Schematic diagram of the components of the semantic category vector of the main object.

[0057] Fig. 9 is the semantic category histogram of the main object.

[0058] Fig.10 This is a schematic diagram of translation change of an embodiment of the present disclosure.

[0059] Fig.11 Schematic diagram of the translation angle between two loop frames according to an embodiment of the present disclosure.

[0060] Fig.12 It is a schematic block diagram of the structure of a semantics-based loop detection device using a hardware processing system according to an embodiment of the present disclosure.

[0061] Description of Reference Numerals

[0062] 1000 Loopback Detection Device

[0063] 1002 Posture Transformation Information Acquisition Module

[0064] 1004 Pose Graph / Map Construction Module

[0065] 1006 Loopback Detection Module

[0066] 1008 Posture Difference Acquisition Module

[0067] 1100 Bus

[0068] 1200 processor

[0069] 1300 Memory

[0070] 1400 Other circuits. DETAILED DESCRIPTION

[0071] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0072] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0073] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0074] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0075] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0076] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0077] First, some technical terms of this disclosure are explained:

[0078] Main object: A semantic object in the scene that is relatively iconic and selected under certain conditions.

[0079] Semantic category vector: The area around the main object is partitioned according to angles, and the distance between the semantic object closest to the main object and the main object in each partition.

[0080] Submap: A local map constructed by multiple consecutive laser frames. Each submap has its own position and multiple submaps can form a larger local map.

[0081] Semantic category histogram: a statistical form of information about semantic objects in the surrounding environment centered on the main object.

[0082] Category label: The category of each laser point after the laser point cloud data passes through the neural network. Each point belongs to one of the result categories of the corresponding neural network.

[0083] Laser frame: The sensor data is stored and transmitted in the form of frames. Each frame of data includes the transmission distance of each point after the sensor emits a laser beam to scan the environment within a fixed time.

[0084] Keyframe: A frame with better quality in the laser frame. It is also used by the backend to construct a pose graph based on the keyframe.

[0085] Coordinate system: The present disclosure preferably uses a rectangular coordinate system, which is defined using a common coordinate system in a robot operating system, and is a right-handed Cartesian rectangular coordinate system (forward is the X axis, left is the Y axis, and upward is the Z axis).

[0086] Posture: The posture involved in the present disclosure is the position and attitude of an object with three degrees of freedom in a two-dimensional plane, where the position is the coordinates in the x-direction and the y-direction, and the attitude is the yaw angle θ with the x-axis as the positive direction.

[0087] Observation position: A rectangular coordinate system is established with the center of the mobile device (autonomous driving vehicle, robot, etc.) as the coordinate origin, the positive direction of the mobile device as the X-axis, and the left as the Y-axis. The position of the main object in the coordinate system is the observation position.

[0088] The sensor data in the present disclosure needs to undergo external parameter conversion of the sensor relative to the center of the mobile device to convert the sensor data to the center of the mobile device. The technical solution of the present disclosure assumes that the sensor data has undergone external parameter conversion. Those skilled in the art should understand that performing external parameter transformation steps before, during or after certain steps of the technical solution of the present disclosure falls within the protection scope of the present disclosure.

[0089] Combined with the following Figures 1 to 12 The semantic-based loop detection method and loop detection device disclosed in the present invention are described in detail.

[0090] Figure 1 It is a flowchart of a semantic-based loop closure detection method according to an embodiment of the present invention.

[0091] refer to Figure 1 , a semantic-based loop detection method S100, comprising:

[0092] S110, acquiring the posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data (i.e., acquiring the posture conversion relationship between adjacent laser frames);

[0093] S120, constructing a pose graph based on at least the pose transformation information between adjacent laser frames, optimizing the pose graph, and constructing a global map based on the optimized pose graph;

[0094] S130, obtaining a preset number of candidate frames (TopN candidate frames) that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculating the similarity between the current frame and each candidate frame, and determining whether a loop occurs.

[0095] A mobile device such as a vehicle continuously acquires laser point cloud data while driving, wherein the multi-frame laser point cloud data described above is laser point cloud data acquired by the mobile device (car, etc.) within a preset time length before the current moment.

[0096] The posture graph is obtained based on the movement trajectory of the mobile device within a preset time length and the posture transformation information between adjacent laser frames.

[0097] According to the preferred embodiment of the present disclosure, the semantic loop closure detection method S100 further includes:

[0098] S140. If a loop occurs, obtain the relative pose error between the two frames where the loop occurs, and use the relative pose error to optimize the pose graph in step S120.

[0099] Figure 3 It is a flowchart for obtaining posture transformation information in a semantic-based loop closure detection method according to an embodiment of the present invention.

[0100] For the semantic-based loop detection method S100 of each of the above embodiments, refer to Figure 3 Preferably, S110, acquiring the pose transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data, comprises:

[0101] S111, performing time synchronization (preferably through timestamps) on the acquired sensor data of SLAM-related sensors including laser radar (including the multi-frame laser point cloud data described above) and performing point cloud distortion compensation on the multi-frame laser point cloud data;

[0102] S112, extracting features from each frame of laser point cloud data to obtain line and surface features of each frame of laser point cloud data;

[0103] S113, performing feature matching calculation on each frame of laser point cloud data to obtain the posture transformation information between two adjacent frames.

[0104] Among them, SLAM-related sensors include one or more of a wheel speed encoder, an inertial measurement unit, a camera, a lidar, a millimeter-wave radar, and a satellite positioning system, and the lidar is used to acquire laser frames.

[0105] Since mobile devices are mostly in motion during the process of collecting laser data, and because the laser frequency is limited, the same frame of laser data is not collected at the same time. The starting laser beam and the ending laser beam of the same laser frame correspond to different positions of the mobile device, so different laser points correspond to different coordinate systems. If the laser point cloud distortion is not considered, there will be large errors in positioning and mapping, so it is necessary to compensate for the point cloud distortion. Point cloud distortion compensation can effectively improve the mapping accuracy.

[0106] According to a preferred embodiment of the present disclosure, point cloud distortion compensation is performed by extrapolating all laser points of the same laser frame to the coordinate system of the starting laser beam (which can be extrapolated according to the motion model). Preferably, the relative motion position of each laser point is obtained based on the motion speed of the mobile device (which can be provided by other sensors in the SLAM-related sensors) and integrated, and then the conversion matrix is ​​calculated based on the relative motion position to convert all laser points to the coordinate system of the initial point.

[0107] For 3D LiDAR, different LiDAR models will produce different single-frame laser point cloud data. However, the common problem is that the amount of point cloud data is large, which puts great pressure on real-time calculation.

[0108] Preferably, each frame of laser point cloud data is sampled to reduce the amount of calculation.

[0109] The sampling method of point cloud data may adopt filtering based on point curvature or filtering based on normal vector variance, etc. In the semantic-related scheme, the size difference of semantic objects may be relatively large. If filtering is performed according to a uniform size, small semantic objects with more features may be filtered out. Therefore, it is preferred to consider adapting the voxel filtering window according to the semantic category to improve the effectiveness of the filtering result.

[0110] Preferably, the 3D space is divided into multiple cubes of fixed volume, a small number of points are retained in each cube according to a preset rule, and the remaining laser points are discarded. The above sampling process can filter out some point cloud noise while optimizing the amount of calculation.

[0111] According to a preferred embodiment of the present disclosure, the line and surface features of each frame of laser point cloud data are obtained based on the smoothness of each point in each frame of laser point cloud data. Preferably, the point cloud is divided into plane points and edge points based on the smoothness. Plane points are points on a smooth plane in three-dimensional space with high smoothness; edge points are points on a sharp edge with low smoothness.

[0112] The above-mentioned posture change information can be obtained through laser odometry, which is the process of using laser radar to infer the posture change of mobile devices. For each laser frame received, point cloud feature matching is performed with its previous frame, and the posture change information between the two frames is calculated through feature matching.

[0113] Among them, point cloud feature matching methods include ICP, NDT, etc. In general, point cloud feature matching is sensitive to the initial value of posture. Therefore, it is preferred to use the output results of other sensors as the initial value, or use the motion model to predict the initial value. The process of calculating and accumulating the posture transformation between frames is the laser odometer. The calculation of the laser odometer is incremental. Therefore, the cumulative error will continue to increase over time, and back-end optimization is required to eliminate the cumulative error.

[0114] Figure 4 This is a process for constructing a pose graph / map in a semantic-based loop closure detection method according to an embodiment of the present invention.

[0115] refer to Figure 4 Preferably, S120 in the semantic-based loop detection method S100 constructs a pose graph based on at least the pose transformation information between adjacent laser frames, optimizes the pose graph, and constructs a global map based on the optimized pose graph, including:

[0116] S121, obtaining at least one key frame in the multi-frame laser point cloud data (which can be obtained based on a preset key frame acquisition standard), taking the key frame pose as a node of a pose graph and optimizing the key frame pose;

[0117] S122. Projecting the laser point cloud of the key frame onto the map based on the optimized key frame pose to obtain a global map.

[0118] In the present disclosure, for all key frames, the key frame poses are taken as nodes of the pose graph, and the pose graph is constructed with spatial constraints as edges of the pose graph. The edges of adjacent nodes represent mileage constraints, and the remaining edges represent closed-loop constraints. When the front end (front-end processing module) continuously sends key frames to the back end (back-end processing module) or a loop is detected, constraints are added. Preferably, the pose graph optimizes the node pose by minimizing the least squares error of the constraints.

[0119] In the present disclosure, after the backend optimizes the key frame pose, the laser point cloud of the key frame is projected onto the map according to the optimized key frame pose to generate a pose graph, and all key frames are projected according to the optimized pose to obtain a global map.

[0120] Figure 5 The present invention is a flowchart of determining a loop frame in a semantic-based loop detection method according to an embodiment of the present invention.

[0121] For the semantic-based loop detection method S100 of each of the above embodiments, preferably, refer to Figure 5 S130, based on semantic information, obtain a preset number of candidate frames (TopN candidate frames) that are most similar to the current frame from the acquired multi-frame laser point cloud data, calculate the similarity between the current frame and each candidate frame, and determine whether a loop occurs, including:

[0122] S131, constructing a global descriptor of the current scene with the main object as the center;

[0123] S132, constructing a semantic category vector of the main object based on the global descriptor and the category and distance of the semantic object closest to the main object;

[0124] S133: Determine the semantic category required for semantic segmentation based on the current scene, and divide the 360-degree range around the main object into N equal parts with the main object as the center. S Equally divided regions, obtaining a semantic category histogram based on the semantic categories of the semantic objects in each of the equally divided regions and the number of semantic objects in each semantic category;

[0125] S134, calculating the semantic category vector similarity between the semantic category vector of the main object in the current frame and the historical semantic category vector (historical semantic category vector) having the same semantic category as the main object in the acquired multi-frame laser point cloud data (i.e., historical key frames) (the same processing is performed for all main objects in the current frame), so as to obtain the historical key frames whose semantic category vector similarity is greater than or equal to the threshold similarity as candidate frames;

[0126] S135 , calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the main object in each candidate frame having the same semantic category as the main object, to obtain the historical key frame with the maximum semantic category histogram similarity.

[0127] Among them, step S134 is a rough loop matching based on the semantic category vector, and step S135 is a fine loop matching based on the semantic category histogram.

[0128] The present invention constructs a scene global descriptor centered on the main object rather than the lidar, which can get rid of the premise that the current frame needs to be very close to the loop frame. As long as the same main object and similar scene global descriptors are detected, the loop can be found and the relative position relationship can be calculated directly based on the semantic descriptor (semantic information), which is more robust and has a higher loop recall rate.

[0129] In more detail, in the same area constructed with the semantic category histogram centered on the main object, the category and distance of the semantic object closest to the main object in each fan-shaped block are calculated to form a semantic category vector of the main object. The dimension of each semantic category vector is equal to N S ,The value of each dimension includes the semantic category and the distance of the main object from the semantic object.

[0130] The mathematical description of the semantic category vector is:

[0131]

[0132] Among them, i represents the i-th main object, N S is the number of sectors divided around the main object, l is the distance of the semantic object closest to the main object in the sector, and t is the semantic category label of the semantic object. The plus sign in the above formula (1) does not represent numerical calculations, but only indicates that the semantic category vector should store both the category of the semantic object and the distance from the main object. After calculating the semantic category vector of each main object, the semantic category vector is stored in the database together with the semantic category of the main object.

[0133] The semantic categories required for semantic segmentation are determined according to the usage scenario (current scene). The main principles for selecting semantic categories include stationary objects in the scene and objects with high differentiation in the scene. Therefore, people and animals will not be considered as semantic categories. Although stationary vehicles have potential mobility, they have high differentiation, so they are included in the semantic categories. At the same time, the vehicle categories are marked and stored separately, and are specially processed in subsequent similarity calculations. After the semantic category is determined, the data set of the corresponding scene is obtained for semantic segmentation training. Semantic segmentation training can be performed based on the RangeNet++ semantic segmentation network. Those skilled in the art can also perform semantic segmentation training based on other types of semantic segmentation networks (such as RandLA-Net, SalsaNet, KPConv, etc.), all of which fall within the scope of protection of the present disclosure.

[0134] When a mobile device, such as a car, is driving, it will continuously acquire laser point cloud information, and multiple laser frames form a submap. For the point cloud information of each submap, a trained semantic segmentation network is used to perform semantic recognition on each point to obtain a semantic category label, and then cluster analysis (such as using the K-Means clustering algorithm) is performed to obtain multiple main objects, and the center of mass of each main object is calculated as the x-coordinate and y-coordinate of the main object.

[0135] Taking the main object as the center, divide the 360-degree range around the main object into N S Preferably, the area around the main object is divided into N equal parts, preferably starting from the connecting line between the main object and the center of the vehicle body, and divided in sequence according to the preset angle intervals, with a division radius of L. S There are multiple semantic objects distributed in each sector. The number of semantic objects is counted according to the semantic category to obtain the semantic category histogram of each sector. The horizontal axis is the category and the vertical axis is the number of the category. The number of semantic categories is N C , and get N S ×N C The semantic category histogram of the main object of length. The mathematical expression of the semantic category histogram is:

[0136]

[0137] Among them, label is the semantic category of the main object, i refers to the histogram of the i-th main object, N C is the total number of semantic categories, N S It is the number of fan-shaped blocks around the main object.

[0138] After calculating the semantic category histogram of each main object, the semantic category histogram is stored in the database together with the semantic category of the main object. So far, each main object corresponds to a semantic category vector and a semantic category histogram.

[0139] From the composition, it can be seen that both the semantic category vector and the semantic category histogram are rotationally invariant, that is, observing from different perspectives will only affect the translation order of the semantic category histogram and the semantic category vector, but will not change the content. This feature ensures the reliability of loop detection.

[0140] Since there are a large number of key frames in laser mapping, the present disclosure needs to perform computational optimization when performing loop detection to speed up loop detection. The present disclosure uses semantic category vectors for coarse matching in order to quickly find the most similar part of the data to the current frame in a large amount of historical key frame data.

[0141] The length of each semantic category vector is N S , each dimension stores both semantic category labels and distance values. When matching, all historical semantic category vectors of the same category as the current main object are retrieved from the database. Taking a single historical semantic category vector as an example, its semantic category vector is q, and the semantic category vector of the main object in the current observation is p. The similarity is first calculated using the category label. Due to the existence of rotation yaw and the rotation invariance of the semantic category vector, rotation is required when calculating the similarity until the semantic categories are completely corresponding or all possible calculations are completed. The calculation formula is:

[0142]

[0143] in, p is the semantic category vector of the main object currently observed, is a single semantic category vector stored in the database. The superscript rt represents the translation change before and after. k is the subscript value of the semantic category vector. n is the number of translation bits when q is translated to have the maximum similarity with p. The translation change can be referred to Fig.10 .

[0144] For the semantic-based loop detection method S100 of the above embodiment, preferably, refer to Figure 6 , step S130 further includes:

[0145] S136. Determine whether the distance between the semantic category histogram of the historical key frame with the largest semantic category histogram similarity and the semantic category histogram of the current frame is less than or equal to the threshold distance. If so, take the historical key frame with the largest semantic category histogram similarity as the loop frame (i.e., loop key frame).

[0146] Step S136 is a loop frame determination step.

[0147] When calculating the histogram similarity described above, Manhattan distance is preferably used. Other formulas may also be used to calculate the histogram distance, such as Euclidean distance or cosine distance, but attention should be paid to increasing the semantic category weight.

[0148] According to a preferred embodiment of the present disclosure, step S134 further includes:

[0149] Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame; and / or,

[0150] Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame.

[0151] In the present disclosure, what needs special treatment is that if the category of a certain position in the semantic category vector is a vehicle, due to the potential mobility of the vehicle, the category may not be completely consistent in the same scene. Therefore, when some positions in the semantic category vector are vehicle labels, the labels of the two semantic category vectors are compared. If the corresponding positions are both vehicles, the semantic category similarity between the two frames is calculated.

[0152] According to a preferred embodiment of the present disclosure, step S134 further includes:

[0153] Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the historical key frame. If so, the vehicle label is not considered when calculating the similarity of the semantic category vector between the current frame and the historical key frame (i.e., the vehicle label is not included in the calculation of the similarity). If not, the vehicle label is considered; and / or,

[0154] Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the historical key frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, the vehicle label is not considered when calculating the similarity of the semantic category vector between the current frame and the historical key frame (that is, the vehicle label is not included in the calculation of the similarity). If not, the vehicle label is considered.

[0155] In this implementation, if the semantic category vector of a main object in the current frame corresponds to the same position as the historical semantic category vector of the main object in the historical key frame except for the vehicle label, but the position of the vehicle is different, it is necessary to further calculate the distance information of the semantic category vector and use the distance value of the corresponding position. If the distance of the category different from the vehicle is less than the distance between the vehicle and the main object, it means that there was no car at that position before, and the category difference is not caused by the movement of the vehicle. In this case, I(p k ,q k )=0; if the distance of the category different from the vehicle is greater than the distance of the vehicle from the main object, it means that the category difference is likely caused by the movement of the vehicle, and the category difference does not affect the same scene. In this case, I(p k ,q k )=1. So far, the historical semantic category vector when the number of translation bits is n has the maximum similarity with the semantic category vector of the current main object, and the maximum similarity is:

[0156]

[0157] According to the preferred embodiment of the present disclosure, after calculating the semantic category vector similarity of all main objects and historical key frames with the same main object category, the threshold ε is used as the criterion for evaluating whether to be filtered:

[0158]

[0159] in, Represents the set of keyframes whose semantic vector similarity is higher than the threshold. If the scene is rich in semantic objects, a larger threshold can be set.

[0160] When the similarity of the semantic category vector of the key frame is higher than the threshold, the rotation angle is saved:

[0161]

[0162] Among them, the left direction is the positive direction, and n is calculated by formula #(4).

[0163] In the semantic category vector matching described above, q rt is the semantic category vector of the rotated keyframe, θ rt is the corresponding rotation angle. The key frames in the corresponding θ rt Rotate the semantic category histograms one by one, and calculate the distance between the rotated histograms and the histogram of the main object:

[0164]

[0165] Among them, h i and k i is the value of the corresponding position in the histogram, ω j is the class weight, j∈{0,1,2...N C}, by defining ω j The importance of each semantic category in the calculation can be controlled. The histogram distance is a representation of the similarity between two histograms. The smaller the distance, the more similar they are.

[0166] For the semantic-based loop detection method S100 of each of the above embodiments, preferably, S135, calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the main object in each candidate frame having the same semantic category as the main object, to obtain the historical key frame with the largest semantic category histogram similarity, includes:

[0167] Determine whether there is a vehicle label in the semantic category vector of the main object in the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the candidate frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the candidate frame. If so, do not include the vehicle-related area in the similarity calculation when calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category. If not, include the vehicle-related area in the similarity calculation; and / or,

[0168] Determine whether there is a vehicle label in the semantic category vector of the main object of the candidate frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the candidate frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, the vehicle-associated area is not included in the similarity calculation when calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category as the main object. If not, the vehicle-associated area is included in the similarity calculation.

[0169] In the precise matching of semantic maps, the impact of vehicle movement on the matching results still needs to be considered. The changes in observation perspective caused by vehicle movement can be divided into two types: one is that there is a vehicle in the historical frame but not in the current frame; the other is that there is no vehicle in the historical frame but there is a vehicle in the current frame. The centroid distance of the semantic object is used when counting the histogram. The volume of the vehicle is generally large, and the observation change caused by it is inversely proportional to its distance from the main object. The closer the distance, the greater the impact. The data stored in the semantic category vector can determine whether the vehicle is the semantic object closest to the main object. If the vehicle is closest to the main object, the fan-shaped block corresponding to the vehicle is not included in the similarity calculation, and only the fan-shaped block without the vehicle is calculated. It should be noted that only the centroid is considered in the calculation, but the actual vehicle may affect more than one fan-shaped block due to its large volume. Therefore, it is necessary to calculate how many fan-shaped blocks (i.e., vehicle-related areas) are affected according to the volume of the vehicle. These affected fan-shaped blocks are not included in the similarity calculation; if the vehicle is not the closest, the result of the corresponding semantic vector is used as the histogram similarity result.

[0170] For the semantic-based loop detection method S100 of each of the above embodiments, preferably, S140, if a loop occurs, obtaining a relative pose error between two frames where the loop occurs, includes:

[0171] For the current frame and the loop frame of the current frame, the pose of the main object in the current frame coordinate system and the loop frame coordinate system is obtained to obtain the relative pose error.

[0172] After calculating the similarity between all candidate loops and the current frame through precise matching, we can get the result with the maximum similarity, that is, the minimum distance. However, the most similar result is not necessarily the loop result, and we need to further determine whether the minimum distance is less than a certain threshold, that is:

[0173] D min ≤δ#(8)

[0174] The historical frame corresponding to the shortest distance that meets the above threshold condition is considered to be a loop frame.

[0175] For the two keyframes that have looped, take out the positions of the main object in the corresponding keyframe coordinate system, which are:

[0176] (x1,y1) and (x2,y2);

[0177] The poses of the two key frames are:

[0178] and

[0179] The derived pose difference of the current key frame pose relative to the loop key frame is:

[0180]

[0181] Among them, θ is the translation angle of rt. For the two frames that are determined to be looped, the relative pose error is calculated and sent to the backend for backend optimization. Fig.11 Schematic diagram of the translation angle between two loop frames according to an embodiment of the present disclosure.

[0182] The semantic-based loop detection method disclosed in the present invention adopts a hierarchical and progressive semantic similarity matching strategy, first performs a coarse match based on the semantic vector (to improve search efficiency), and then performs a fine match based on the semantic histogram. During the matching process, it supports the processing of movable obstacles such as vehicles, eliminates scene changes caused by movable obstacles, and can improve the success rate of loop detection in scenes with more car movement. The semantic-based loop detection method disclosed in the present invention can directly calculate the relative position relationship between the two loop frames.

[0183] Figure 7 It is the semantic object of a single main object and the entire fan-shaped area around it, that is, all the components of the semantic histogram.

[0184] Figure 8 yes Figure 7 The constituent elements of the semantic category vector of the main object in , that is, the semantic object closest to the main object in each fan-shaped block.

[0185] Fig. 9 It is the semantic category histogram of the main object, where each fan-shaped block counts the number of semantic objects of all semantic categories.

[0186] According to another aspect of the present disclosure, a semantics-based loop closure detection device is provided.

[0187] According to an embodiment of the present disclosure, a semantic-based loop closure detection device 1000 includes:

[0188] A posture transformation information acquisition module 1002, which acquires posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data (i.e., acquires posture conversion relationship between adjacent laser frames);

[0189] A pose graph / map construction module 1004, the pose graph / map construction module 1004 constructs a pose graph based on at least the pose transformation information between adjacent laser frames, optimizes the pose graph, and constructs a global map based on the optimized pose graph;

[0190] The loop detection module 1006 obtains a preset number of candidate frames (TopN candidate frames) that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculates the similarity between the current frame and each candidate frame, and determines whether a loop occurs.

[0191] Preferably, the semantic-based loop detection device 1000 of the present disclosure further includes: a posture difference acquisition module 1008. If a loop occurs, the posture difference acquisition module 1008 acquires the relative posture error between the two frames where the loop occurs.

[0192] The semantic-based loop detection device of each of the above-mentioned embodiments can be implemented in the form of a computer software architecture.

[0193] For the semantic-based loop detection device 1000 of each of the above-mentioned embodiments, preferably, it also includes: a sensor device, which is at least used to obtain laser point cloud data.

[0194] The technical solution disclosed in the present invention uses a sensor device on a mobile device (autonomous driving vehicles, robots, etc.) to perform a SLAM (simultaneous localization and mapping) process during the movement of the mobile device, including front-end odometer, back-end optimization, loop detection and real-time mapping.

[0195] Mobile devices include autonomous vehicles, robots and other autonomous mobile devices that require real-time mapping and positioning with distance detection sensors installed. Distance detection sensors include lidar, depth cameras, millimeter wave radar and other sensors that can obtain environmental distance information.

[0196] Fig.12 It is a schematic block diagram of the structure of a semantics-based loop detection device using a hardware processing system according to an embodiment of the present disclosure.

[0197] The loop detection device may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by the corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0198] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0199] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0200] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0201] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0202] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0203] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0204] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0205] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0206] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned semantic-based loop detection method.

[0207] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned semantic-based loop detection method.

[0208] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0209] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0210] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A semantic-based loop detection method, characterized in that: include: S110, acquiring position and posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data; S120, constructing a pose graph based at least on pose transformation information between adjacent laser frames, optimizing the pose graph, and constructing a global map based on the optimized pose graph; as well as S130, acquiring a preset number of candidate frames that are most similar to the current frame from the acquired multiple frames of laser point cloud data based on semantic information, calculating the similarity between the current frame and each candidate frame, and determining whether a loop occurs; Among them, S130, based on the semantic information, obtaining a preset number of candidate frames that are most similar to the current frame from the acquired multi-frame laser point cloud data, calculating the similarity between the current frame and each candidate frame, and determining whether a loop occurs, includes: S131, constructing a global descriptor of the current scene with the main object as the center; S132, constructing a semantic category vector of the main object based on the global descriptor and the category and distance of the semantic object closest to the main object; S133: Determine the semantic category required for semantic segmentation based on the current scene, and divide the 360-degree range around the main object into equal parts with the main object as the center. Equally divided regions, obtaining a semantic category histogram based on the semantic categories of the semantic objects in each of the equally divided regions and the number of semantic objects in each semantic category; S134, calculating the semantic category vector similarity between the semantic category vector of the main object in the current frame and the historical semantic category vectors having the same semantic category as the main object in the acquired multi-frame laser point cloud data, so as to obtain the historical key frames whose semantic category vector similarity is greater than or equal to the threshold similarity as the candidate frames; S135, calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the main object in each candidate frame, so as to obtain the historical key frame with the maximum semantic category histogram similarity; S136, determining whether the distance between the semantic category histogram of the historical key frame with the largest semantic category histogram similarity and the semantic category histogram of the current frame is less than or equal to a threshold distance; if so, taking the historical key frame with the largest semantic category histogram similarity as a loop frame.

2. The semantic-based loop detection method according to claim 1, characterized in that: Also includes: S140. If a loop occurs, obtain a relative pose error between the two frames where the loop occurs, and use the relative pose error to optimize the pose graph in step S120.

3. The semantic-based loop detection method according to claim 1, characterized in that: S110, acquiring the pose transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data, including: S111, performing time synchronization on the acquired sensor data of SLAM related sensors including laser radar and performing point cloud distortion compensation on multiple frames of laser point cloud data; S112, extracting features from each frame of laser point cloud data to obtain line and surface features of each frame of laser point cloud data; and S113, performing feature matching calculation on each frame of laser point cloud data to obtain the posture transformation information between two adjacent frames.

4. The semantic-based loop detection method according to claim 1 or 2, characterized in that: S120, constructing a pose graph based at least on pose transformation information between adjacent laser frames, optimizing the pose graph, and constructing a global map based on the optimized pose graph, including: S121, obtaining at least one key frame in the multiple frames of laser point cloud data, taking the key frame pose as a node of a pose graph and optimizing the key frame pose; and S122. Projecting the laser point cloud of the key frame onto the map based on the optimized key frame pose to obtain a global map.

5. The semantic-based loop detection method according to claim 1, characterized in that: Step S134 also includes: Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame; and / or Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If so, calculate the semantic category vector similarity between the current frame and the historical key frame.

6. The semantic-based loop detection method according to claim 5, characterized in that: Step S134 also includes: Determine whether there is a vehicle label in the semantic category vector of the main object of the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the historical key frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the historical key frame. If so, the vehicle label is not considered when calculating the semantic category vector similarity between the current frame and the historical key frame. If not, the vehicle label is considered; and / or Determine whether there is a vehicle label in the semantic category vector of the main object of the historical key frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the historical key frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, the vehicle label is not considered when calculating the similarity of the semantic category vector between the current frame and the historical key frame. If not, the vehicle label is considered.

7. The semantic-based loop detection method according to claim 6, characterized in that: S135, calculating the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the main object in each candidate frame, so as to obtain the historical key frame with the largest semantic category histogram similarity, including: Determine whether there is a vehicle label in the semantic category vector of the main object in the current frame. If so, determine whether the corresponding position label in the historical semantic category vector of the candidate frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the semantic category vector of the main object in the current frame is less than the distance between the main object and the corresponding position label in the historical semantic category vector of the main object in the candidate frame. If so, calculate the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category as the main object without including the vehicle-related area in the similarity calculation. If not, include the vehicle-related area in the similarity calculation; and / or Determine whether there is a vehicle label in the semantic category vector of the main object of the candidate frame. If so, determine whether the corresponding position label in the semantic category vector of the current frame is a vehicle label. If not, determine whether the distance between the main object and the vehicle label in the historical semantic category vector of the main object in the candidate frame is less than the distance between the main object and the corresponding position label in the semantic category vector of the main object in the current frame. If so, calculate the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histogram of the main object in the candidate frame with the same semantic category as the main object without including the vehicle-related area in the similarity calculation. If not, include the vehicle-related area in the similarity calculation.

8. The semantic-based loop detection method according to claim 7, characterized in that: S140: If a loop occurs, obtain the relative position error between the two frames where the loop occurs, including: For the current frame and the loop frame of the current frame, the pose of the main object in the current frame coordinate system and the loop frame coordinate system is obtained to obtain the relative pose error.

9. A semantic-based loop detection device, characterized in that: include: A posture transformation information acquisition module, which acquires posture transformation information between adjacent laser frames based on the acquired multi-frame laser point cloud data; A pose graph / map construction module, wherein the pose graph / map construction module constructs a pose graph based on at least pose transformation information between adjacent laser frames, optimizes the pose graph, and constructs a global map based on the optimized pose graph; as well as A loop detection module, wherein the loop detection module obtains a preset number of candidate frames that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculates the similarity between the current frame and each candidate frame, and determines whether a loop occurs; Wherein, obtaining a preset number of candidate frames that are most similar to the current frame from the acquired multi-frame laser point cloud data based on semantic information, calculating the similarity between the current frame and each candidate frame, and determining whether a loop occurs, including: Construct a global descriptor of the current scene centered on the main object; Construct the semantic category vector of the main object based on the global descriptor and the category and distance of the semantic object closest to the main object; Determine the semantic categories required for semantic segmentation based on the current scene, and divide the 360-degree range around the main object into equal parts with the main object as the center. Equally divided regions, obtaining a semantic category histogram based on the semantic categories of the semantic objects in each of the equally divided regions and the number of semantic objects in each semantic category; Calculating the semantic category vector similarity between the semantic category vector of the main object in the current frame and the historical semantic category vectors having the same semantic category as the main object in the acquired multi-frame laser point cloud data, so as to obtain historical key frames whose semantic category vector similarity is greater than or equal to a threshold similarity as candidate frames; Calculate the similarity between the semantic category histogram of the main object in the current frame and the historical semantic category histograms of the same semantic category as the main object in each candidate frame to obtain the historical key frame with the maximum semantic category histogram similarity; It is determined whether the distance between the semantic category histogram of the historical key frame with the largest semantic category histogram similarity and the semantic category histogram of the current frame is less than or equal to the threshold distance. If so, the historical key frame with the largest semantic category histogram similarity is used as the loop frame.

10. The semantic-based loop detection device according to claim 9, characterized in that: Also includes: A posture difference acquisition module, if a loop occurs, the posture difference acquisition module obtains the relative posture error between the two frames where the loop occurs.

11. The semantic-based loop detection device according to claim 10, characterized in that: Also includes: A sensor device is used at least to acquire laser point cloud data.

12. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory so that the processor executes the method according to any one of claims 1 to 8.

13. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images