Multi-sensor fusion SLAM data generation method and device, SLAM equipment and electronic equipment

Abstract

The invention discloses a multi-sensor fusion SLAM (Simultaneous Localization and Mapping) data generation method and device, SLAM equipment and electronic equipment. The method comprises the following steps: constructing a first pose error according to a deviation between a gravity vector and an accelerometer measurement value under a local coordinate system, calculating first pose compensation data, and superposing the first pose compensation data with first inertial data to obtain compensated second inertial data; updating a pose based on the second inertial data, and obtaining pose data of the first environment image frame sequence under the global coordinate system; and calculating an optimized pose based on the first pose data and / or the second pose data, and associating the optimized pose with the global pose of the image frame to generate a pose constraint factor. Performing loopback detection on the second environment image frame sequence to obtain a loopback constraint factor; and carrying out joint modeling on the visual odometer factor, the laser odometer factor, the inertial measurement unit pre-integration factor, the image frame global pose and the loop constraint factor, and executing nonlinear optimization to generate SLAM data. Geometric consistency verification is carried out on visual feature matching through compensated inertial data, so that the reliability and precision of the front-end visual odometer can be improved; and visual loopback detection and laser loopback detection based on scanning context descriptors are complementarily fused, so that the robustness and the constraint reliability of loopback retrieval are improved, and the positioning and mapping precision and the global consistency of the SLAM system in a complex environment are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of three-dimensional imaging technology, and in particular to a multi-sensor fusion SLAM method and apparatus, as well as SLAM devices and electronic equipment. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a key technology for autonomous mobile robots in large-scale map building. Especially in outdoor environments, mobile robots typically rely on LiDAR (LiDAR) sensors to perform SLAM tasks for high-precision localization and navigation. To overcome the limitations of single-sensor solutions, researchers have proposed various multi-sensor fusion SLAM methods. Common multi-sensor fusion architectures include vision-inertial fusion, LiDAR-inertial fusion, and a combination of vision, LiDAR, and inertial data. Existing technologies that integrate data from multiple sensors can enhance the performance of autonomous mobile platforms such as mobile robots and unmanned vehicles in unknown or semi-unknown environments for autonomous localization and environmental mapping.

[0003] Existing multi-sensor fusion SLAM technologies still face challenges in terms of pose estimation stability, visual feature matching reliability, and loop closure detection accuracy in complex environments. Therefore, a technical solution is needed to improve the localization and mapping accuracy and global consistency of vision-laser fusion SLAM systems. Summary of the Invention

[0004] This application provides a method and apparatus for generating SLAM data through multi-sensor fusion, as well as SLAM devices and electronic devices, to address the shortcomings of existing technologies in terms of low accuracy and overall global consistency.

[0005] To achieve the above objectives, this application provides a multi-sensor fusion SLAM data generation method, applied to a SLAM device. The SLAM device includes a visual odometry, a laser odometry, and an inertial measurement unit. The method is characterized by comprising: Acquire multi-sensor data, which includes a first environmental image frame sequence acquired by the visual odometry and first pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, first point cloud data acquired by the laser odometry and second pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, and first inertial data acquired by the inertial measurement unit. The first inertial data is integrated to obtain the third pose data; The first pose error is constructed based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and the first pose compensation data is calculated. The first pose compensation data is then superimposed with the first inertial data to form the compensated second inertial data. The third pose data is updated using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system; Image features are extracted from the image frames in the first environmental image frame sequence to determine feature points, and the first relative pose between adjacent image frames is determined based on feature point matching calculation; Based on the fourth pose data of each image frame, calculate the predicted position of the feature point in the previous image frame in the next frame of the adjacent image frames. Based on the deviation between the actual and predicted positions of feature points in the next frame, feature points with deviations exceeding a preset threshold are identified as error feature points to obtain the second environmental image frame sequence. The fifth pose data is calculated based on the first pose data and / or the second pose data, and the pose constraint factor is generated by associating the fourth pose data of each image frame with the fifth pose data in the global coordinate system. Loop closure detection is performed on each image frame in the second environmental image frame sequence to identify image frames whose similarity to the preceding historical image frames meets a preset condition as loop closure image frames, and a loop closure constraint factor is calculated based on the loop closure image frames. The first pose data is used as the visual odometry factor, the second pose data as the laser odometry factor, and the first relative pose as the inertial measurement unit pre-integration factor. Based on the visual odometry factor, laser odometry factor, inertial measurement unit pre-integration factor, fourth pose data, and closure constraint factor, nonlinear optimization is performed on the fifth pose data to generate SLAM data.

[0006] In the SLAM data generation method according to embodiments of this application, the first inertial data includes angular velocity data, and the step of integrating the first inertial data to obtain the third pose data includes: Integrating the first angular velocity data yields discrete-time attitude quaternions, which serve as the third pose data. The calculation of the first pose compensation data, which involves superimposing the first pose compensation data with the first inertial data to form the compensated second inertial data, includes: The angular velocity compensation is calculated by proportional-integral adjustment and used as the first pose compensation data. The angular velocity compensation is then superimposed on the original angular velocity data to obtain the third pose data.

[0007] According to the SLAM data generation method of this application embodiment, before performing integration processing on the first inertial data, the method further includes: The first pose data, the first point cloud data, and the first inertial data are synchronized in time. The external calibration parameters between the visual odometer, the laser odometer, and the inertial measurement unit are configured.

[0008] According to the SLAM data generation method of this application embodiment, the loop closure detection for each image frame in the second environment image frame sequence includes: For each image frame, loop closure detection is performed using both visual loop closure detection and laser loop closure detection. The visual loop closure detection includes calculating the similarity between the feature descriptors of the current image frame and the historical image frames based on the bag-of-words model, and retrieving candidate visual loop closure frames from the historical image frames whose similarity exceeds a first threshold. The laser loop closure detection includes determining the descriptor of each image frame based on the first point cloud data using a scanning context method, and determining the image frame whose similarity to the preceding historical image frame meets a preset condition as a loop closure image frame includes: Based on the descriptors of the current image frame and the historical image frames, the similarity between the current image frame and the historical image frames is calculated, and the current image frame and the corresponding historical image frame with a similarity less than a preset second threshold are identified as loopback image frames.

[0009] According to the SLAM data generation method of this application embodiment, the step of determining the descriptor of each image frame based on the first point cloud data using a scanning context method includes: Within the XOY plane of the point cloud coordinate system of the first point cloud data, centered on the origin of the coordinate system, the area is divided circumferentially into... A sector-shaped region, further divided radially into The first point cloud data is divided into multiple cells by a ring, wherein the width of each cell is as shown in expression (1). (1) in, The radial width of the cell. This is the corner width of the cell. This represents the maximum detection range of the lidar. Calculate the cell value according to the expression (2) below. (2) in, Indicates the first The first ring The value of a cell in a sector, It is the set of all points in that cell. yes The Z-axis coordinate of the point; Construct a dimension using the value of each cell. The matrix is ​​used as the laser loop descriptor for the current image frame.

[0010] In the SLAM data generation method according to the embodiments of this application, the laser loop closure detection includes: For each image frame in each image frame, the loop bond k of that image frame is calculated using the following expression (3) based on the laser loop closure descriptor of the current image frame. (3) Where k is a value of size A one-dimensional array; A binary tree is constructed using the cyclic key k of the image frame to filter out multiple candidate historical image frames from the historical image frames of that image frame; Use the following expression (4) to calculate the similarity distance between the descriptor of the current image frame and the candidate historical image frames. (4) in, For column offset, Representation descriptor matrix The List, The cosine distance between two column vectors: (5) When the similarity distance D is less than a preset second threshold, the current image frame and the corresponding historical image frame are determined as loopback image frames.

[0011] According to the SLAM data generation method of this application embodiment, the step of integrating the first angular velocity data to obtain the attitude quaternion at discrete time points as the third pose data includes: The first angular velocity data is pre-integrated to obtain the following expression (6) representing the change in angular velocity. (6) in The quaternion representation of attitude transformation. This refers to quaternion multiplication. This is the first angular velocity; For expression (6), use the following expression (7) to obtain the third pose data. (7) in for The attitude quaternion at a given moment; Furthermore, the proportional-integral function used in the proportional-integral adjustment process is shown in the following expression (8). (8) in, This is the proportionality coefficient. The integral coefficient is... This is the angular velocity compensation value. The error value is shown in the expression (9) below. (9) in, This refers to the gravity vector in the device's body coordinate system. and These are the normalized vectors of gravity and acceleration measurements, respectively. Let be the rotation matrix from the global coordinate system to the body coordinate system, and as shown in the expression (10) below. (10).

[0012] This application also provides a multi-sensor fusion SLAM data generation device, applied to a SLAM device, the SLAM device including a visual odometry, a laser odometry, and an inertial measurement unit, and further comprising: The acquisition module is used to acquire multi-sensor data, which includes a first environmental image frame sequence acquired by the visual odometry and first pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, first point cloud data acquired by the laser odometry and second pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, and first inertial data acquired by the inertial measurement unit. An integration processing module is used to integrate the first inertial data to obtain the third pose data; The first calculation module is used to construct the first pose error based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and to calculate the first pose compensation data, so as to superimpose the first pose compensation data with the first inertial data to form the compensated second inertial data. The first update module is used to update the third pose data using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. The second calculation module is used to extract image features from the image frames in the first environmental image frame sequence to determine feature points, and to determine the first relative pose between adjacent image frames based on feature point matching calculation; to calculate the predicted position of the feature points in the previous image frame in the next frame in the adjacent image frames according to the fourth pose data of each image frame; to determine the feature points whose deviation exceeds a preset threshold as error feature points according to the deviation between the actual position and the predicted position of the feature points in the next frame, so as to obtain the second environmental image frame sequence; to calculate the fifth pose data according to the first pose data and / or the second pose data, and to generate a pose constraint factor by associating the fourth pose data of each image frame in the global coordinate system with the fifth pose data; The loop closure detection module is used to perform loop closure detection on each image frame in the second environmental image frame sequence, so as to identify the image frame whose similarity with the historical image frame preceding it meets the preset condition as the loop closure image frame, and calculate the loop closure constraint factor based on the loop closure image frame. The generation module is used to take the first pose data as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit pre-integration factor. Based on the visual odometry factor, laser odometry factor, inertial measurement unit pre-integration factor, fourth pose data, and closure constraint factor, it performs nonlinear optimization on the fifth pose data to generate SLAM data.

[0013] This application also provides a SLAM device, including: The system includes a lidar, a camera, an inertial measurement unit (IMU), and a processing unit, wherein the lidar, the camera, and the IMU are respectively connected to the processing unit. The camera is used to acquire a first environmental image frame sequence and calculate the first pose data corresponding to each image frame in the first environmental image frame sequence; The lidar is used to acquire first point cloud data and calculate second pose data corresponding to each image frame in the first environmental image frame sequence. The inertial measurement unit is used to acquire first inertial data; The processing unit is used to integrate the first inertial data to obtain third pose data; construct a first pose error based on the deviation between the gravity vector and the accelerometer measurement in the local coordinate system, and calculate the first pose compensation data, so as to superimpose the first pose compensation data with the first inertial data to form compensated second inertial data; update the third pose data using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system; extract image features from the image frames in the first environmental image frame sequence to determine feature points, and determine the first relative pose between adjacent image frames based on feature point matching calculation; calculate the predicted position of the feature point in the previous image frame in the next frame in the adjacent image frame based on the fourth pose data of each image frame; and determine the first relative pose between adjacent image frames based on the deviation between the actual position and the predicted position of the feature point in the next frame and the deviation exceeding a preset threshold. The feature points of the values ​​are determined as error feature points to obtain a second environmental image frame sequence; the fifth pose data is calculated based on the first pose data and / or the second pose data, and the fourth pose data of each image frame in the global coordinate system is associated with the fifth pose data to generate a pose constraint factor; loop closure detection is performed on each image frame in the second environmental image frame sequence to determine the image frame whose similarity with the historical image frame preceding it meets the preset condition as a loop closure image frame, and the loop closure constraint factor is calculated based on the loop closure image frame; the first pose data is used as a visual odometry factor, the second pose data is used as a laser odometry factor, the first relative pose is used as an inertial measurement unit pre-integration factor, and nonlinear optimization is performed based on the fifth pose data according to the visual odometry factor, laser odometry factor, inertial measurement unit pre-integration factor, fourth pose data, and loop closure constraint factor to generate SLAM data.

[0014] This application also provides an electronic device, including: Memory, used to store programs; A processor is configured to run the program stored in the memory, wherein the program executes the multi-sensor fusion SLAM data generation method provided in the embodiments of this application.

[0015] This application also provides a computer-readable storage medium storing a computer program executable by a processor, wherein when the program is executed by the processor, it implements the multi-sensor fusion SLAM data generation as provided in this application embodiment.

[0016] The multi-sensor fusion SLAM method, apparatus, SLAM device, electronic device, and storage medium provided in this application can construct a first pose error based on the deviation between the gravity vector and the accelerometer measurement value in the local coordinate system, and calculate the first pose compensation data. The first pose compensation data is then superimposed with the first inertial data to form compensated second inertial data. The second inertial data is used to update the third pose data, obtaining the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. The fifth pose data is calculated based on the first pose data and / or the second pose data, and the fourth pose data and fifth pose data of each image frame in the global coordinate system are correlated to generate a pose constraint factor. Loop closure detection is performed on each image frame in the second environmental image frame sequence to link each image frame with its preceding historical frames. Image frames whose similarity between historical image frames meets preset conditions are identified as loop closure image frames, and loop closure constraint factors are calculated based on these image frames. The first pose data is used as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit (IMU) pre-integration factor. Based on the visual odometry factor, laser odometry factor, IMU pre-integration factor, fourth pose data, and loop closure constraint factor, nonlinear optimization is performed on the fifth pose data to generate SLAM data. Therefore, based on the pose data updated using compensated inertial data, the inter-frame transformation matrix between adjacent image frames is calculated, and the visual feature point matching results are geometrically consistent. This allows for timely removal of incorrect or large-deviation matches, improving the reliability and accuracy of the front-end visual odometry results, and overall enhancing the positioning accuracy and stability of the SLAM data. Furthermore, in the loop closure detection section, this application introduces an independently operating laser loop closure detection process on top of visual loop closure detection. By using a scanning context descriptor to process video frames, the impact of viewpoint changes on laser loop closure detection is effectively reduced, improving the efficiency and robustness of loop closure candidate retrieval. By combining visual loop closure detection and laser loop closure detection, the advantages of different sensors in terms of appearance and geometric information can be fully utilized to obtain more reliable loop closure constraint factors. This significantly improves the localization and mapping accuracy of multi-sensor fusion SLAM systems in complex environments and enhances the consistency of global trajectory and map.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of the SLAM data generation method based on multi-sensor fusion according to this application. Figure 2 This is a schematic diagram illustrating the division of cells using the scanning context method in the multi-sensor fusion SLAM data generation method of this application embodiment; Figure 3 This is a schematic diagram of an embodiment of the multi-sensor fusion SLAM data generation device according to this application. Figure 4 This is a schematic diagram of the structure of an embodiment of the SLAM device according to this application; Figure 5 A schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0020] When autonomous mobile platforms such as mobile robots and unmanned vehicles operate in unknown or semi-unknown environments, they typically need to possess autonomous localization and environmental mapping capabilities, i.e., Simultaneous Localization and Mapping (SLAM). As a crucial foundational technology in the field of robot perception and navigation, the localization accuracy and system robustness of SLAM technology directly affect the safety and reliability of robot path planning, obstacle avoidance control, and task execution.

[0021] In existing technologies, SLAM systems mostly rely on a single sensor or a small number of sensors to complete environmental perception and pose estimation. Taking visual SLAM as an example, it continuously acquires image sequences through a camera, extracts and matches image features, estimates the relative pose between adjacent image frames, thereby reconstructing the robot's motion trajectory and generating a dense or sparse environmental map. For LiDAR SLAM, it uses continuously acquired point cloud data for scanning matching or point cloud registration to calculate the pose transformation between adjacent point cloud frames, achieving mapping and localization of two-dimensional or three-dimensional scenes.

[0022] However, SLAM solutions based on a single sensor have significant limitations. Visual SLAM is highly sensitive to factors such as ambient lighting, scene texture, and field-of-view occlusion. In situations with drastic lighting changes, limited texture, or highly dynamic environments, feature matching errors or tracking interruptions can easily occur, leading to a rapid accumulation of pose estimation errors and even system failure. While LiDAR SLAM is not dependent on lighting conditions, it is also prone to registration errors or getting trapped in local optima in confined indoor spaces, scenes with mirrors or glass reflections, and large areas of repetitive structures. Furthermore, single-vision or LiDAR odometry methods often use forward cumulative integration to estimate robot pose; as runtime increases, accumulated errors and trajectory drift become increasingly prominent.

[0023] To improve the overall accuracy and robustness of SLAM systems, existing technologies propose multi-sensor fusion schemes that jointly process data from multiple sources, including cameras, lidar, and inertial measurement units (IMUs). IMUs can output angular velocity and linear acceleration information at high frequencies, which can be used to infer attitude and motion states in a short time, providing some compensation when visual or lidar information is temporarily lost or degraded. Some existing technologies employ IMU pre-integration methods, integrating IMU measurements between adjacent keyframes and incorporating the resulting constraints as factors into graph optimization or filtering frameworks to mitigate the cumulative drift in pose estimation.

[0024] However, while existing technologies utilize IMU data to assist in pose estimation, they lack an effective filtering mechanism based on geometric relationships during the visual front-end feature matching stage. False matches are difficult to eliminate in a timely manner, leading to significant fluctuations in the output quality of visual odometry. Furthermore, existing loop closure detection methods largely rely on a single visual bag-of-words model or indicators based on laser scan similarity. When there are significant changes in viewpoint, obvious lighting variations, or highly repetitive local structures in the scene, loop closure detection is prone to missed or false detections, weakening the global consistency effect of factor graph optimization.

[0025] Example 1 To address the problems of the prior art, this application provides a multi-sensor fusion SLAM data generation method. Figure 1 A schematic flowchart illustrating a multi-sensor fusion SLAM data generation method according to an embodiment of this application is shown. Figure 1 As shown in the embodiments of this application, the multi-sensor fusion SLAM data generation method can be applied to a SLAM device, which may include a visual odometry, a laser odometry, and an inertial measurement unit, and the method may include: S101 acquires data from multiple sensors.

[0026] In step S101, data acquired by multiple sensors of the SLAM device can be obtained as multi-sensor data. For example, a first environmental image frame sequence can be acquired by a visual odometry of a camera and the first pose data corresponding to each image frame in the first environmental image frame sequence can be calculated. A first point cloud data can be acquired by a laser odometry and the second pose data corresponding to each image frame in the first environmental image frame sequence can be calculated. And first inertial data can be acquired by an inertial measurement unit.

[0027] S102, Integrate the first inertial data to obtain the third pose data.

[0028] S103, construct the first attitude error based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and calculate the first attitude compensation data, so as to superimpose the first attitude compensation data with the first inertial data to form the compensated second inertial data.

[0029] S104, update the third pose data using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system.

[0030] According to the embodiments of this application, the first inertial data can be integrated in step S102 to obtain the third pose data. In step S103, the first pose error is constructed based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and the first pose compensation data is calculated. The first pose compensation data is then superimposed with the first inertial data to form the compensated second inertial data. In step S104, the third pose data is updated using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. In particular, the inertial measurement unit may include a gyroscope and an accelerometer, and angular velocity data can be obtained through the gyroscope. Thus, in step S102, the angular velocity data can be integrated to obtain the attitude quaternion at discrete moments as the third pose data. For example, the first angular velocity data can be pre-integrated to obtain the following expression (6) representing the change in angular velocity. (6) in The quaternion representation of attitude transformation. This refers to quaternion multiplication. This is the first angular velocity; For expression (6), use the following expression (7) to obtain the third pose data. (7) in for The attitude quaternion at a given moment; Furthermore, the angular velocity compensation can be calculated using proportional-integral (PI) adjustment processing as the first pose compensation data. This angular velocity compensation is then superimposed onto the original angular velocity data to obtain the third pose data. The proportional-integral function used in the PI adjustment processing can be expressed as shown in the following expression (8). (8) in, This is the proportionality coefficient. The integral coefficient is... This is the angular velocity compensation value. The error value is shown in the expression (9) below. (9) in, This refers to the gravity vector in the device's body coordinate system. and These are the normalized vectors of gravity and acceleration measurements, respectively. Let be the rotation matrix from the global coordinate system to the body coordinate system, and as shown in the expression (10) below. (10).

[0031] S105, extract image features from the image frames in the first environmental image frame sequence to determine feature points, and determine the first relative pose between adjacent image frames based on feature point matching calculation.

[0032] S106, based on the fourth pose data of each image frame, calculate the predicted position of the feature point in the previous image frame in the next frame of the adjacent image frames.

[0033] S107, based on the deviation between the actual position and the predicted position of the feature points in the next frame, feature points with deviations exceeding a preset threshold are identified as error feature points, so as to obtain the second environmental image frame sequence.

[0034] In steps S105 and S106, image features can be extracted from each image frame in the first environmental image frame sequence obtained in step S101 to determine feature points. Based on feature point matching, the first relative pose between adjacent image frames is determined. According to the fourth pose data of each image frame, the predicted position of the feature point in the preceding image frame in the following frame is calculated. In step S107, based on the deviation between the actual and predicted positions of the feature points in the following frame, feature points with deviations exceeding a preset threshold are identified as error feature points, thus obtaining the second environmental image frame sequence. Therefore, feature point matching can promptly eliminate incorrect or large-deviation matches, improving the reliability and accuracy of the front-end visual odometry results and enhancing the overall positioning accuracy and stability of SLAM data.

[0035] S108, calculate the fifth pose data based on the first pose data and / or the second pose data, and associate the fourth pose data of each image frame with the fifth pose data in the global coordinate system to generate a pose constraint factor.

[0036] In step S108, the fifth pose data can be calculated based on the first pose data and / or the second pose data, and the fourth pose data and the fifth pose data of each image frame in the global coordinate system are correlated to generate pose constraint factors. For example, in this embodiment, a factor graph can be used, which consists of nodes and edges. Nodes include the pose of the device at different times and the observed landmarks, with the observation of landmarks represented by keyframes. Edges connect the nodes and represent various factors, including the pose calculated by visual odometry and laser odometry, the relative pose change between two keyframes obtained by IMU pre-integration, and the loop closure detection factor. As a nonlinear optimization method, factor graph optimization can find a set of node values ​​that minimize the sum of squared errors of all factors. In this embodiment, the absolute pose of the device in the global coordinate system at each time point obtained by complementary filtering of IMU data can be added to the factor graph as a new constraint factor. Compared with the relative pose between frames, the absolute pose can better constrain the overall optimization range of factor graph optimization and reduce excessive overall trajectory changes caused by over-optimization.

[0037] S109, perform loop closure detection on each image frame in the second environmental image frame sequence, so as to determine the image frame whose similarity with the historical image frame preceding it meets the preset condition as the loop closure image frame, and calculate the loop closure constraint factor based on the loop closure image frame.

[0038] In step S109, loop closure detection can be performed on each image frame in the second environmental image frame sequence generated in step S107, so that the image frames whose similarity with the historical image frames preceding them meets the preset conditions are identified as loop closure image frames, and the loop closure constraint factor is calculated based on the loop closure image frames.

[0039] For example, in the embodiments of this application, visual loop closure detection and laser loop closure detection can be used for each image frame. Visual loop closure detection includes calculating the similarity between the feature descriptors of the current image frame and historical image frames based on the bag-of-words model, and retrieving candidate visual loop closure frames from historical image frames whose similarity exceeds a first threshold. Laser loop closure detection includes determining the descriptor of each image frame based on the first point cloud data using a scanning context method.

[0040] Furthermore, based on the descriptors of the current image frame and the historical image frames, the similarity between the current image frame and the historical image frames can be calculated, and the current image frame and the corresponding historical image frame with a similarity less than a preset second threshold can be identified as loopback image frames.

[0041] Figure 2 This is a schematic diagram illustrating the division of cells using the scanning context method in the multi-sensor fusion SLAM data generation method of this application embodiment.

[0042] like Figure 2 As shown in the SLAM data generation method according to the embodiments of this application, the XOY plane of the point cloud coordinate system of the first point cloud data can be divided circumferentially with the origin of the coordinate system as the center. A sector-shaped region, further divided radially into The first point cloud data is divided into multiple cells by a ring, where the width of each cell is as shown in expression (1). (1) in, The radial width of the cell. This is the corner width of the cell. This represents the maximum detection range of the lidar. Calculate the cell value according to the expression (2) below. (2) in, Indicates the first The first ring The value of a cell in a sector, It is the set of all points in that cell. yes The Z-axis coordinate of the point; Construct a dimension using the value of each cell. The matrix is ​​used as the laser loop descriptor for the current image frame.

[0043] In the SLAM data generation method according to embodiments of this application, laser loop closure detection may include: For each image frame in each image frame, the loop bond k of that image frame is calculated using the following expression (3) based on the laser loop closure descriptor of the current image frame. (3) Where k is a value of size A one-dimensional array; A binary tree is constructed using the cyclic key k of the image frame to filter out multiple candidate historical image frames from the historical image frames of that image frame; Use the following expression (4) to calculate the similarity distance between the descriptor of the current image frame and the candidate historical image frames. (4) in, For column offset, Representation descriptor matrix The List, The cosine distance between two column vectors: (5) When the similarity distance D is less than the preset second threshold, the current image frame and the corresponding historical image frame are determined as loopback image frames.

[0044] S110: The first pose data is used as the visual odometry factor, the second pose data is used as the laser odometry factor, the first relative pose is used as the inertial measurement unit pre-integration factor, and nonlinear optimization is performed based on the fifth pose data according to the visual odometry factor, laser odometry factor, inertial measurement unit pre-integration factor, fourth pose data, and closure constraint factor to generate SLAM data.

[0045] Finally, in step S110, the first pose data can be used as the visual odometry factor, the second pose data as the laser odometry factor, and the first relative pose as the inertial measurement unit pre-integration factor. Based on the visual odometry factor, the laser odometry factor, the inertial measurement unit pre-integration factor, the fourth pose data, and the loop closure constraint factor, nonlinear optimization is performed on the fifth pose data to generate SLAM data.

[0046] Therefore, the multi-sensor fusion SLAM data generation method provided in this application can construct a first pose error based on the deviation between the gravity vector and the accelerometer measurement value in the local coordinate system, and calculate the first pose compensation data. The first pose compensation data is then superimposed with the first inertial data to form compensated second inertial data. The second inertial data is used to update the third pose data, obtaining the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. The fifth pose data is calculated based on the first pose data and / or the second pose data, and the fourth pose data and fifth pose data of each image frame in the global coordinate system are correlated to generate a pose constraint factor. Loop closure detection is performed on each image frame in the second environmental image frame sequence to link each image frame with the preceding historical image frames. Image frames whose similarity meets preset conditions are identified as loop closure image frames, and loop closure constraint factors are calculated based on these image frames. The first pose data is used as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit (IMU) pre-integration factor. Based on the visual odometry factor, laser odometry factor, IMU pre-integration factor, fourth pose data, and loop closure constraint factor, nonlinear optimization is performed on the fifth pose data to generate SLAM data. Therefore, based on the pose data updated using compensated inertial data, the inter-frame transformation matrix between adjacent image frames is calculated, and the visual feature point matching results are geometrically consistent. This allows for timely removal of incorrect or large-deviation matches, improving the reliability and accuracy of the front-end visual odometry results, and overall enhancing the positioning accuracy and stability of the SLAM data. Furthermore, in the loop closure detection section, this application introduces an independently operating laser loop closure detection process on top of visual loop closure detection. By using a scanning context descriptor to process video frames, the impact of viewpoint changes on laser loop closure detection is effectively reduced, improving the efficiency and robustness of loop closure candidate retrieval. By combining visual loop closure detection and laser loop closure detection, the advantages of different sensors in terms of appearance and geometric information can be fully utilized to obtain more reliable loop closure constraint factors. This significantly improves the localization and mapping accuracy of multi-sensor fusion SLAM systems in complex environments and enhances the consistency of global trajectory and map.

[0047] Example 2 Figure 3 This is a schematic diagram of an embodiment of the multi-sensor fusion SLAM data generation device according to this application. Figure 3As shown in the illustration, this application embodiment also provides a multi-sensor fusion SLAM data generation device, which is applied to a SLAM device. The SLAM device includes a visual odometry, a laser odometry, and an inertial measurement unit. The SLAM data generation device includes an acquisition module 201, an integration processing module 202, a first calculation module 203, a first update module 204, a second calculation module 205, a loop closure detection module 206, and a generation module 207.

[0048] The acquisition module 201 can be used to acquire multi-sensor data, which may include a first environmental image frame sequence acquired by the visual odometry and first pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, first point cloud data acquired by the laser odometry and second pose data corresponding to each image frame in the first environmental image frame sequence calculated by the visual odometry, and first inertial data acquired by the inertial measurement unit.

[0049] The integration processing module 202 can be used to integrate the first inertial data to obtain the third pose data.

[0050] The first calculation module 203 can be used to construct the first attitude error based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and calculate the first attitude compensation data, so as to superimpose the first attitude compensation data with the first inertial data to form the compensated second inertial data.

[0051] The first update module 204 can be used to update the third pose data using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system.

[0052] According to the embodiments of this application, the integration processing module 202 can perform integration processing on the first inertial data to obtain the third pose data. The first calculation module 203 can construct the first pose error based on the deviation between the gravity vector in the local coordinate system and the accelerometer measurement value, and calculate the first pose compensation data. The first pose compensation data is then superimposed on the first inertial data to form the compensated second inertial data. The first update module 204 can use the second inertial data to update the third pose data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. In particular, the inertial measurement unit may include a gyroscope and an accelerometer, and angular velocity data can be obtained through the gyroscope. Thus, the integration processing module 202 can integrate the angular velocity data to obtain the attitude quaternion at discrete moments as the third pose data. For example, the first angular velocity data is pre-integrated to obtain the following expression (6) representing the change in angular velocity. (6) in The quaternion representation of attitude transformation. This refers to quaternion multiplication. This is the first angular velocity; The integration processing module 202 can use the following expression (7) to obtain the third pose data for expression (6). (7) in for The attitude quaternion at a given moment; Furthermore, the first calculation module 203 can calculate the angular velocity compensation amount as the first pose compensation data through proportional-integral adjustment processing, and then superimpose the angular velocity compensation amount onto the original angular velocity data to obtain the third pose data. The proportional-integral function used in the proportional-integral adjustment processing can be shown in the following expression (8). (8) in, This is the proportionality coefficient. The integral coefficient is... This is the angular velocity compensation value. The error value is shown in the following expression (9). (9) in, This refers to the gravity vector in the device's body coordinate system. and These are the normalized vectors of gravity and acceleration measurements, respectively. Let be the rotation matrix from the global coordinate system to the body coordinate system, and as shown in the expression (10) below. (10).

[0053] The second calculation module 205 can be used to extract image features from the image frames in the first environmental image frame sequence to determine feature points, and to determine the first relative pose between adjacent image frames based on feature point matching calculation; to calculate the predicted position of the feature points in the previous image frame in the next frame according to the fourth pose data of each image frame; to determine the feature points with deviations exceeding a preset threshold as error feature points according to the deviation between the actual position and the predicted position of the feature points in the next frame, so as to obtain the second environmental image frame sequence; to calculate the fifth pose data according to the first pose data and / or the second pose data, and to generate a pose constraint factor by associating the fourth pose data of each image frame in the global coordinate system with the fifth pose data.

[0054] The second calculation module 205 can extract image features from each image frame in the first environmental image frame sequence acquired by the acquisition module 201 to determine feature points, and determine the first relative pose between adjacent image frames based on feature point matching calculation. Based on the fourth pose data of each image frame, it calculates the predicted position of the feature points in the previous image frame in the next frame. Based on the deviation between the actual and predicted positions of the feature points in the next frame, feature points with deviations exceeding a preset threshold are identified as error feature points, thus obtaining the second environmental image frame sequence. Therefore, feature point matching can be used to promptly eliminate incorrect or large-deviation matches, improving the reliability and accuracy of the front-end visual odometry results, and overall enhancing the positioning accuracy and stability of SLAM data.

[0055] The second calculation module 205 can calculate the fifth pose data based on the first pose data and / or the second pose data, and associate the fourth pose data and the fifth pose data of each image frame in the global coordinate system to generate pose constraint factors. For example, in this embodiment, a factor graph can be used, which consists of nodes and edges. Nodes include the pose of the device at different times and the observed landmarks, with the observation of landmarks represented by keyframes. Edges connect the nodes and represent various factors, including the pose calculated by visual odometry and laser odometry, the relative pose change between two keyframes obtained by IMU pre-integration, and the loop closure detection factor. As a nonlinear optimization method, factor graph optimization can find a set of node values ​​that minimize the sum of squared errors of all factors. In this embodiment, the absolute pose of the device in the global coordinate system at each time point obtained by complementary filtering of IMU data can be added to the factor graph as a new constraint factor. Compared with the relative pose between frames, the absolute pose can better constrain the overall optimization range of factor graph optimization and reduce excessive overall trajectory changes caused by over-optimization.

[0056] The loop closure detection module 206 can be used to perform loop closure detection on each image frame in the second environmental image frame sequence, so as to determine the image frame whose similarity with the historical image frame preceding it meets the preset conditions as the loop closure image frame, and calculate the loop closure constraint factor based on the loop closure image frame.

[0057] The loop closure detection module 206 can perform loop closure detection on each image frame in the second environmental image frame sequence generated by the second calculation module 205, so as to determine the image frame whose similarity with the historical image frame preceding it meets the preset condition as the loop closure image frame, and calculate the loop closure constraint factor based on the loop closure image frame.

[0058] For example, in this embodiment, the loop closure detection module 206 can perform loop closure detection for each image frame using visual loop closure detection and laser loop closure detection. Visual loop closure detection includes calculating the similarity between the feature descriptors of the current image frame and historical image frames based on the bag-of-words model, and retrieving candidate visual loop closure frames from historical image frames whose similarity exceeds a first threshold. Laser loop closure detection includes determining the descriptors of each image frame based on the first point cloud data using a scanning context method.

[0059] Furthermore, the loop closure detection module 206 can further calculate the similarity between the current image frame and the historical image frame based on the descriptors of the current image frame and the historical image frame, and determine the current image frame and the corresponding historical image frame with a similarity less than a preset second threshold as loop closure image frames.

[0060] The loop closure detection module 206 can divide the XOY plane of the point cloud coordinate system of the first point cloud data, with the origin of the coordinate system as the center, along the circumferential direction into... A sector-shaped region, further divided radially into The first point cloud data is divided into multiple cells by a ring, where the width of each cell is as shown in expression (1). (1) in, The radial width of the cell. This is the corner width of the cell. This represents the maximum detection range of the lidar. Calculate the cell value according to the expression (2) below. (2) in, Indicates the first The first ring The value of a cell in a sector, It is the set of all points in that cell. yes The Z-axis coordinate of the point; Construct a dimension using the value of each cell. The matrix is ​​used as the laser loop descriptor for the current image frame.

[0061] In the SLAM data generation method according to embodiments of this application, laser loop closure detection may include: For each image frame in each image frame, the loop bond k of that image frame is calculated using the following expression (3) based on the laser loop closure descriptor of the current image frame. (3) Where k is a value of size A one-dimensional array; A binary tree is constructed using the cyclic key k of the image frame to filter out multiple candidate historical image frames from the historical image frames of that image frame; Use the following expression (4) to calculate the similarity distance between the descriptor of the current image frame and the candidate historical image frames. (4) in, For column offset, Representation descriptor matrix The List, The cosine distance between two column vectors: (5) When the similarity distance D is less than the preset second threshold, the current image frame and the corresponding historical image frame are determined as loopback image frames.

[0062] The generation module 207 can be used to take the first pose data as a visual odometry factor, the second pose data as a laser odometry factor, the first relative pose as an inertial measurement unit pre-integration factor, and perform nonlinear optimization based on the fifth pose data according to the visual odometry factor, laser odometry factor, inertial measurement unit pre-integration factor, fourth pose data, and closure constraint factor to generate SLAM data.

[0063] Therefore, the multi-sensor fusion SLAM data generation apparatus provided in this application embodiment can construct a first pose error based on the deviation between the gravity vector and the accelerometer measurement value in the local coordinate system, and calculate the first pose compensation data. The first pose compensation data is then superimposed with the first inertial data to form compensated second inertial data. The second inertial data is used to update the third pose data, obtaining the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system. The fifth pose data is calculated based on the first pose data and / or the second pose data, and the fourth pose data and fifth pose data of each image frame in the global coordinate system are correlated to generate a pose constraint factor. Loop closure detection is performed on each image frame in the second environmental image frame sequence to link each image frame with the preceding historical image frames. Image frames whose similarity meets preset conditions are identified as loop closure image frames, and loop closure constraint factors are calculated based on these image frames. The first pose data is used as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit (IMU) pre-integration factor. Based on the visual odometry factor, laser odometry factor, IMU pre-integration factor, fourth pose data, and loop closure constraint factor, nonlinear optimization is performed on the fifth pose data to generate SLAM data. Therefore, based on the pose data updated using compensated inertial data, the inter-frame transformation matrix between adjacent image frames is calculated, and the visual feature point matching results are geometrically consistent. This allows for timely removal of incorrect or large-deviation matches, improving the reliability and accuracy of the front-end visual odometry results, and overall enhancing the positioning accuracy and stability of the SLAM data. Furthermore, in the loop closure detection section, this application introduces an independently operating laser loop closure detection process on top of visual loop closure detection. By using a scanning context descriptor to process video frames, the impact of viewpoint changes on laser loop closure detection is effectively reduced, improving the efficiency and robustness of loop closure candidate retrieval. By combining visual loop closure detection and laser loop closure detection, the advantages of different sensors in terms of appearance and geometric information can be fully utilized to obtain more reliable loop closure constraint factors. This significantly improves the localization and mapping accuracy of multi-sensor fusion SLAM systems in complex environments and enhances the consistency of global trajectory and map.

[0064] Example 3 Figure 4 This is a schematic diagram illustrating the structure of an embodiment of the SLAM device according to this application. Figure 4 As shown in the illustration, the SLAM device provided in this application embodiment includes: a lidar 31, a camera 32, an inertial measurement unit 33, and a processing unit 34. The lidar 31, camera 32, and inertial measurement unit 33 are respectively connected to the processing unit 34.

[0065] Camera 32 can be used to acquire a first environmental image frame sequence and calculate the first pose data corresponding to each image frame in the first environmental image frame sequence.

[0066] The lidar 31 can be used to acquire first point cloud data and calculate second pose data corresponding to each image frame in the first environmental image frame sequence.

[0067] The inertial measurement unit 33 is used to acquire the first inertial data.

[0068] The processing unit 34 can be used to implement the SLAM data generation method according to the embodiments of this application. For example, it can be used to integrate the first inertial data to obtain the third pose data; construct the first pose error based on the deviation between the gravity vector and the accelerometer measurement value in the local coordinate system, and calculate the first pose compensation data, so as to superimpose the first pose compensation data with the first inertial data to form the compensated second inertial data; update the third pose data using the second inertial data to obtain the fourth pose data of each image frame in the first environmental image frame sequence in the global coordinate system; extract image features from the image frames in the first environmental image frame sequence to determine feature points, and determine the first relative pose between adjacent image frames based on feature point matching calculation; calculate the predicted position of the feature point in the previous image frame in the next frame in the adjacent image frame based on the fourth pose data of each image frame; and calculate the predicted position of the feature point in the next frame based on the actual position and predicted position of the feature point in the next frame. The system calculates the first pose data and / or the second pose data, and associates the fourth pose data of each image frame with the fifth pose data in the global coordinate system to generate a pose constraint factor. It then performs loop closure detection on each image frame in the second environment image frame sequence, identifying image frames whose similarity to previous historical image frames meets a preset condition as loop closure image frames, and calculates a loop closure constraint factor based on these image frames. Finally, it uses the first pose data as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit (IMU) pre-integration factor. Based on the visual odometry factor, laser odometry factor, IMU pre-integration factor, fourth pose data, and loop closure constraint factor, it performs nonlinear optimization on the fifth pose data to generate SLAM data.

[0069] Example 5 The above describes the internal functions and structure of the multi-sensor fusion SLAM method, which can be implemented as an electronic device. Figure 5 A schematic diagram illustrating the structure of an embodiment of the electronic device provided in this application. (See attached diagram.) Figure 5 As shown, the electronic device includes a memory 41 and a processor 42.

[0070] Memory 41 is used to store programs. In addition to the programs described above, memory 41 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0071] The memory 41 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0072] Processor 42 is not limited to a processor (CPU), but may also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural network processor (NPU), or an artificial intelligence (AI) chip. Processor 42 is coupled to memory 31 and executes the program stored in memory 31 to perform the multi-sensor fusion SLAM data generation method of Embodiment 1 described above.

[0073] Furthermore, such as Figure 5 As shown, the electronic device may also include other components such as a communication component 43, a power supply component 44, an audio component 4, and a display 46. Figure 5 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 5 The components shown.

[0074] Communication component 43 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 3G, 4G, or 5G, or combinations thereof. In one exemplary embodiment, communication component 33 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 43 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0075] Power supply component 44 provides power to various components of the electronic device. Power supply component 44 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0076] Audio component 45 is configured to output and / or input audio signals. For example, audio component 45 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 41 or transmitted via communication component 43. In some embodiments, audio component 45 also includes a speaker for outputting audio signals.

[0077] Display 46 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0078] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for multi-sensor fusion SLAM data generation, applied to a SLAM device, the SLAM device comprising a visual odometry, a laser odometry and an inertial measurement unit, characterized in that, The method comprises: acquiring multi-sensor data, the multi-sensor data comprising a first sequence of environment image frames collected by the visual odometry and first pose data corresponding to each image frame in the first sequence of environment image frames calculated by the visual odometry, first point cloud data collected by the laser odometry and second pose data corresponding to each image frame in the first sequence of environment image frames calculated by the visual odometry, and first inertial data collected by the inertial measurement unit; integrating the first inertial data to obtain third pose data; constructing a first pose error according to a deviation between a gravity vector in a local coordinate system and an accelerometer measurement value, and calculating first pose compensation data to superimpose the first pose compensation data and the first inertial data to form compensated second inertial data; updating the third pose data using the second inertial data to obtain fourth pose data of each image frame in the first sequence of environment image frames in a global coordinate system; extracting image features from the image frames in the first sequence of environment image frames to determine feature points, and determining a first relative pose between adjacent image frames based on feature point matching calculation; calculating a predicted position of a feature point in a previous image frame in adjacent image frames according to the fourth pose data of each image frame; determining, according to a deviation between an actual position and a predicted position of a feature point in a subsequent frame, a feature point whose deviation exceeds a preset threshold as an error feature point to obtain a second sequence of environment image frames; calculating fifth pose data according to the first pose data and / or the second pose data, and correlating the fourth pose data of each image frame in the global coordinate system and the fifth pose data to generate a pose constraint factor; performing loop detection on each image frame in the second sequence of environment image frames to determine an image frame between which a similarity between each image frame and a historical image frame located before the each image frame satisfies a preset condition as a loop image frame, and calculating a loop constraint factor according to the loop image frame; taking the first pose data as a visual odometry factor, taking the second pose data as a laser odometry factor, taking the first relative pose as an inertial measurement unit pre-integration factor, and performing nonlinear optimization based on the fifth pose data according to the visual odometry factor, the laser odometry factor, the inertial measurement unit pre-integration factor, the fourth pose data, and the loop constraint factor to generate SLAM data.

2. The SLAM data generation method of claim 1, wherein, The first inertial data comprises angular velocity data, and the integrating the first inertial data to obtain third pose data comprises: integrating the first angular velocity data to obtain a discrete-time attitude quaternion as the third pose data, and the calculating first pose compensation data to superimpose the first pose compensation data and the first inertial data to form compensated second inertial data comprises: calculating an angular velocity compensation amount as the first pose compensation data through proportional integral adjustment processing to superimpose the angular velocity compensation amount to original angular velocity data to obtain the third pose data.

3. The SLAM data generation method of claim 1, before the integrating the first inertial data, the method further comprises: time synchronizing the first pose data, the first point cloud data and the first inertial data; configuring external calibration parameters between the visual odometry, the laser odometry and the inertial measurement unit.

4. The SLAM data generation method of claim 1, wherein, the loop detection for each image frame in the second sequence of environment image frames comprises: using visual loop detection and laser loop detection for each image frame, the visual loop detection comprises similarity calculation of feature descriptors of a current image frame and historical image frames based on a bag-of-words model, and retrieving candidate visual loop frames from the historical image frames whose similarity exceeds a first threshold value; the laser loop detection comprises determining descriptors of each image frame based on the first point cloud data using a scan context method, and determining an image frame as a loop image frame if similarity between the image frame and a historical image frame located before the image frame satisfies a preset condition comprises: calculating similarity between the current image frame and the historical image frame based on the descriptors of the current image frame and the historical image frame, and determining the current image frame and the corresponding historical image frame as a loop image frame if the similarity is less than a preset second threshold value.

5. The SLAM data generation method of claim 4, wherein, the determining descriptors of each image frame based on the first point cloud data using a scan context method comprises: In the XOY plane of the point cloud coordinate system of the first point cloud data, the XOY plane is divided into a plurality of sector regions with the coordinate system origin as the center along the circumferential direction, and the sector regions are further divided into a plurality of rings along the radial direction, and the first point cloud data is divided into a plurality of unit cells, wherein the width of each unit cell is shown as expression (1) （1） wherein, is the radial width of the cell, is the angular width of the cell, is the maximum detection range of the lidar; calculating a value of a single cell according to the following expression (2) (2) in, Indicates the first The first ring The value of a cell in a sector, It is the set of all points in that cell. yes The Z-axis coordinate of the point; A matrix of dimension is constructed using the values of each cell as the laser loop descriptor for the current image frame.

6. The SLAM data generating method of claim 5, wherein, the laser loop detection comprises: for each image frame in each image frame, calculating a loop key k of the image frame using the following expression (3) according to the laser loop descriptors of the current image frame (3) where k is a one-dimensional array of size k = 2. constructing a binary tree using the loop key k of the image frame to filter a plurality of candidate historical image frames from the historical image frames of the image frame; calculating a similarity distance between the descriptors of the current image frame and the candidate historical image frames using the following expression (4) (4) wherein, is a column offset, denotes the descriptor matrix of the column, is the cosine distance between two column vectors: (5) when the similarity distance D is less than a preset second threshold value, determining the current image frame and the corresponding historical image frame as a loop image frame.

7. The SLAM data generating method of claim 2, wherein, the integrating the first angular velocity data to obtain a discrete-time attitude quaternion as the third pose data comprises: pre-integrating the first angular velocity data to obtain an angular velocity change amount represented by the following expression (6) (6) wherein is a quaternion representation of the pose transform, is a quaternion multiplication operation, is the first angular velocity; obtaining the third pose data using the following expression (7) for expression (6) (7) wherein is the pose quaternion at the time instant and the proportional integral function used in the proportional integral adjustment processing is shown in the following expression (8) (8) wherein, is a proportional coefficient, is an integral coefficient, is an angular velocity compensation value, is an error value and is shown in the following expression (9) (9) wherein is the gravity vector in the body coordinate system of the device, and are normalized vectors of the gravity and acceleration measurements, respectively, is the rotation matrix from the global coordinate system to the body coordinate system and is given by expression (10) below, (10)。 8.A multi-sensor fusion SLAM data generation apparatus applied to a SLAM device, the SLAM device comprising a visual odometry, a laser odometry and an inertial measurement unit, characterized in that, comprises: an acquisition module, configured to acquire multi-sensor data, the multi-sensor data comprising a first sequence of environment image frames collected by the visual odometry and first pose data corresponding to each image frame in the first sequence of environment image frames calculated by the visual odometry, first point cloud data collected by the laser odometry and second pose data corresponding to each image frame in the first sequence of environment image frames calculated by the visual odometry, and first inertial data collected by the inertial measurement unit; an integration processing module, configured to integrate the first inertial data to obtain third pose data; The first computing module is configured to construct a first pose error according to a deviation between a gravity vector in a local coordinate system and an accelerometer measurement value, and to calculate first pose compensation data, so as to superimpose the first pose compensation data on the first inertial data to form second inertial data after compensation; The first updating module is configured to update the third pose data by using the second inertial data to obtain fourth pose data of each image frame in the first environment image frame sequence in a global coordinate system; The second computing module is configured to extract image features from the image frames in the first environment image frame sequence to determine feature points, and to determine a first relative pose between adjacent image frames based on feature point matching calculation; to calculate a predicted position of a feature point in a previous image frame in adjacent image frames in a subsequent frame according to the fourth pose data of each image frame; to determine a feature point whose deviation between an actual position and a predicted position exceeds a preset threshold as an error feature point according to the deviation between the actual position and the predicted position of the feature point in the subsequent frame, so as to obtain a second environment image frame sequence; to calculate fifth pose data according to the first pose data and / or the second pose data, and to generate a pose constraint factor by correlating the fourth pose data of each image frame in the global coordinate system and the fifth pose data; The loop detection module is configured to perform loop detection on each image frame in the second environment image frame sequence, to determine an image frame between which a similarity between each image frame and a historical image frame located before the each image frame satisfies a preset condition as a loop image frame, and to calculate a loop constraint factor according to the loop image frame; The generation module is configured to take the first pose data as a visual odometry factor, to take the second pose data as a laser odometry factor, to take the first relative pose as an inertial measurement unit pre-integration factor, and to perform nonlinear optimization based on the fifth pose data according to the visual odometry factor, the laser odometry factor, the inertial measurement unit pre-integration factor, the fourth pose data, and the loop constraint factor, so as to generate SLAM data.

9. A SLAM device, characterized by It comprises: a laser radar, a camera, an inertial measurement unit, and a processing unit, wherein the laser radar, the camera, and the inertial measurement unit are respectively connected to the processing unit, and the camera is configured to collect a first environment image frame sequence and to calculate first pose data corresponding to each image frame in the first environment image frame sequence; the laser radar is configured to obtain first point cloud data and to calculate second pose data corresponding to each image frame in the first environment image frame sequence; the inertial measurement unit is configured to obtain first inertial data; and the processing unit is configured to perform the following steps: a first computing module is configured to construct a first pose error according to a deviation between a gravity vector in a local coordinate system and an accelerometer measurement value, and to calculate first pose compensation data, so as to superimpose the first pose compensation data on the first inertial data to form second inertial data after compensation; a first updating module is configured to update the third pose data by using the second inertial data to obtain fourth pose data of each image frame in the first environment image frame sequence in a global coordinate system; a second computing module is configured to extract image features from the image frames in the first environment image frame sequence to determine feature points, and to determine a first relative pose between adjacent image frames based on feature point matching calculation; to calculate a predicted position of a feature point in a previous image frame in adjacent image frames in a subsequent frame according to the fourth pose data of each image frame; to determine a feature point whose deviation between an actual position and a predicted position exceeds a preset threshold as an error feature point according to the deviation between the actual position and the predicted position of the feature point in the subsequent frame, so as to obtain a second environment image frame sequence; to calculate fifth pose data according to the first pose data and / or the second pose data, and to generate a pose constraint factor by correlating the fourth pose data of each image frame in the global coordinate system and the fifth pose data; a loop detection module is configured to perform loop detection on each image frame in the second environment image frame sequence, to determine an image frame between which a similarity between each image frame and a historical image frame located before the each image frame satisfies a preset condition as a loop image frame, and to calculate a loop constraint factor according to the loop image frame; a generation module is configured to take the first pose data as a visual odometry factor, to take the second pose data as a laser odometry factor, to take the first relative pose as an inertial measurement unit pre-integration factor, and to perform nonlinear optimization based on the fifth pose data according to the visual odometry factor, the laser odometry factor, the inertial measurement unit pre-integration factor, the fourth pose data, and the loop constraint factor, so as to generate SLAM data. The processing unit is configured to integrate the first inertial data to obtain third pose data, construct a first pose error according to a deviation between a gravity vector in a local coordinate system and an accelerometer measurement value, and calculate first pose compensation data, and superimpose the first pose compensation data on the first inertial data to form compensated second inertial data, update the third pose data using the second inertial data to obtain fourth pose data of each image frame in the first sequence of environment image frames in a global coordinate system, extract image features from the image frames in the first sequence of environment image frames to determine feature points, and determine a first relative pose between adjacent image frames based on feature point matching calculation, calculate a predicted position of a feature point in a previous image frame in adjacent image frames in the fourth pose data of each image frame, determine a feature point whose deviation between an actual position and a predicted position exceeds a preset threshold as an error feature point according to the deviation between the actual position and the predicted position of the feature point in the subsequent frame, to obtain a second sequence of environment image frames, calculate fifth pose data according to the first pose data and / or the second pose data, and associate the fourth pose data of each image frame in the global coordinate system with the fifth pose data to generate a pose constraint factor, perform loop detection on each image frame in the second sequence of environment image frames to determine an image frame that satisfies a preset condition in similarity between each image frame and a historical image frame located before it as a loop image frame, and calculate a loop constraint factor according to the loop image frame, take the first pose data as a visual odometry factor, the second pose data as a laser odometry factor, and the first relative pose as an inertial measurement unit pre-integration factor, and perform nonlinear optimization based on the fifth pose data according to the visual odometry factor, the laser odometry factor, the inertial measurement unit pre-integration factor, the fourth pose data, and the loop constraint factor, to generate SLAM data.

10. An electronic device, comprising: Comprise: a memory for storing a program; a processor for running the program stored in the memory to execute the multi-sensor fusion SLAM data generation method of any one of claims 1-7.

Images