Joint mapping method and device based on multi-sensor fusion, equipment and medium

Abstract

The invention relates to a joint mapping method, device and equipment based on multi-sensor fusion, and a medium. The method comprises the following steps: obtaining image data, 2D laser radar data and IMU data, and carrying out data preprocessing; in combination with the IMU data, applying a visual and IMU fused SLAM algorithm to each visual frame of the image data, and estimating the pose at each moment; performing time binding on the visual frame and a laser frame in the 2D laser radar data based on the timestamp; performing motion distortion correction on 2D laser radar data based on IMU data, performing temporal pose interpolation on data of a current laser frame in combination with pose estimation at each moment, and performing spatial binding on the laser frame and a visual frame; and carrying out local matching on the interpolated laser radar data by adopting a laser scanning matching algorithm, and constructing a 2D map. Compared with the prior art, the method has the advantages of low cost and the like.

Description

Technical Field

[0001] The present invention relates to the field of robot mapping, and in particular to a joint mapping method, device, equipment and medium based on multi-sensor fusion. Background Art

[0002] Traditional 2D LiDARs are often unable to construct effective 2D grid maps in some complex scenarios due to the observation range and reflection angle limitations of the laser beam. Especially in open outdoor areas, the measurement accuracy and coverage of 2D LiDARs are limited, making it difficult to accurately capture the features of objects at long distances or with high differences. In addition, complex slope terrain can also cause large errors in the data scanned by LiDARs. In these scenarios, traditional 2D LiDARs often cannot provide sufficient and effective map information.

[0003] 3D laser radar can provide more detailed and high-precision three-dimensional maps, and can effectively overcome the limitations of 2D laser radar in complex scenes. For example, CN117367427A discloses a multi-modal SLAM technology for visual-assisted laser odometers suitable for indoor environments such as warehouses. The specific steps are: first, the system motion is estimated using images and laser point clouds, the feature tracking module detects visual feature points, and the depth map registration module aligns the local depth map and point cloud to obtain the depth of the visual feature points. The frame-to-frame motion estimation module uses feature points to calculate the body motion. The laser radar data is used to extract plane and edge features. With the data obtained by the visual module as a priori, the extracted features and IMU measurements are input into the state estimation module, and the state estimation is performed at 10Hz-50Hz. The estimated posture registers the feature points into the global frame and merges them with the constructed feature point map. The updated map will add more new points in the next step. This solution fuses visual features with 3D radar point clouds, and further uses IMU for Kalman filtering fusion, and finally provides a more stable posture output to solve the problem of 3D radar failure. However, 3D LiDAR is expensive and bulky, making it difficult to meet the needs of low-cost, compact robots. In addition, for most robot applications, especially in large open spaces, it is usually only necessary to generate an accurate 2D plane map for positioning and navigation.

[0004] Some existing solutions use pose estimation based on visual SLAM to directly interpolate all laser frames into a 2D map, trying to achieve the fusion of visual and lidar data through the pose information provided by visual SLAM. Although this method is simple, it often leads to obvious edge ghosting and jaggedness in the map in practical applications. A better solution is to separate the 2D lidar and the visual sensor to build the map synchronously, and the visual sensor uses the visual SLAM solution to provide a visual odometer as the input of the laser SLAM system. The map constructed by this solution is greatly improved compared with the former, but the problem is that the visual SLAM and laser SLAM are not synchronized in scale and coordinate system, resulting in the inability to directly connect and fuse the maps they generate, limiting the effective application of maps in positioning and navigation. Secondly, in some complex scenes, especially in open spaces or environments lacking obvious landmarks, the amount of information of 2D lidar is usually less than that of visual information, making closed-loop detection more difficult in larger-scale environments. For example, CN114608561B discloses a positioning and mapping method and system based on multi-sensor fusion. Although this method solves the problem of closed-loop detection difficulties of 2D lidar, it does not consider the synchronization problem between the visual sensor and the lidar in scale and coordinate system, and the complex data fusion process also increases the computing cost and reduces the mapping efficiency.

[0005] Therefore, how to make full use of the information of multiple sensors (such as vision, IMU, 2D lidar) at a low cost, overcome the limitations of a single sensor, and achieve efficient and accurate 2D plane map construction has become a difficult problem that needs to be solved urgently in the current technical field. Summary of the invention

[0006] The purpose of the present invention is to overcome the limitations of traditional 2D lidar that it cannot effectively construct 2D plane maps in complex environments, as well as the problems of coordinate system asynchrony and closed-loop difficulty in existing vision and laser synchronous mapping solutions, and to provide a joint mapping method, device, equipment and medium based on multi-sensor fusion. By fusing the data of vision, IMU and lidar, efficient, accurate and low-cost multi-sensor fusion mapping is achieved, which can generate accurate 2D plane maps in complex environments to meet the needs of positioning and navigation.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] According to a first aspect of the present invention, a joint mapping method based on multi-sensor fusion is provided, the method comprising the following steps:

[0009] S1, acquire image data, 2D lidar data and IMU data and perform data preprocessing;

[0010] S2, combined with IMU data, applies the SLAM algorithm of vision and IMU fusion to each visual frame of image data to estimate the pose at each moment;

[0011] S3, temporally binds the visual frame and the laser frame in the 2D lidar data based on the timestamp;

[0012] S4, based on the IMU data, performs motion distortion correction on the 2D lidar data, and combines the pose estimation at each moment to perform temporal pose interpolation on the data of the current laser frame, and spatially binds the laser frame and the visual frame;

[0013] S5, using the laser scanning matching algorithm to perform local matching on the interpolated lidar data to construct a 2D map.

[0014] As a preferred technical solution, the S2 comprises the following steps:

[0015] S21, feature extraction: extracting feature points from the image through a feature extraction algorithm;

[0016] S22, feature matching: matching the extracted feature points with the existing map point cloud, and accurately estimating the relative motion of the camera between two frames in combination with IMU data;

[0017] S23, pose estimation: solve the camera pose based on the feature matching results.

[0018] As an optimal technical solution, the S3 is specifically as follows: based on each point in the lidar frame, according to its relative timestamp in the frame, its absolute timestamp on the global time axis is determined, and based on the timestamp of the visual frame and the absolute timestamp of the lidar frame, the time series is aligned to complete the time binding.

[0019] As a preferred technical solution, the posture interpolation is specifically as follows: for the translation part, linear interpolation is used to obtain an estimated value, for the rotation part, spherical linear interpolation is used to obtain an estimated value, and the estimated values ​​of the translation part and the rotation part are combined to obtain the final posture interpolation.

[0020] As a preferred technical solution, in step S4, the acceleration and angular velocity data provided by the IMU data are used to compensate for the angular error in the LiDAR frame pose in real time, and the motion distortion compensation is performed on each frame of LiDAR data through the IMU data.

[0021] As a preferred technical solution, when the visual odometer is lost and unable to obtain image data due to changes in lighting, occlusion or other factors, a 2D lidar is used as a supplementary sensor to provide a laser odometer in real time. Through the 2D lidar data, the robot's position can be stably tracked in a short time until the visual odometer resumes normal operation.

[0022] As a preferred technical solution, the method further includes: while updating the 2D grid map, constructing a 3D visual map in real time, the two are performed synchronously and coordinated through an optimization algorithm.

[0023] According to a second aspect of the present invention, a joint mapping device based on multi-sensor fusion is provided, comprising:

[0024] Data acquisition module: acquires image data, 2D lidar data and IMU data and performs data preprocessing;

[0025] Preliminary pose estimation module: Combined with IMU data, the SLAM algorithm combining vision and IMU fusion is applied to each visual frame of the image data to estimate the pose at each moment;

[0026] Time binding module: Temporally binds the visual frames and the laser frames in the 2D lidar data based on the timestamp;

[0027] Spatial binding module: It performs motion distortion correction on 2D lidar data based on IMU data, and interpolates the pose of the current laser frame in time based on the pose estimation at each moment, and spatially binds the laser frame and the visual frame.

[0028] Mapping module: The laser scanning matching algorithm is used to perform local matching on the interpolated lidar data to construct a 2D map.

[0029] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0030] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1. Efficient multi-sensor data fusion

[0033] The present invention uses precise multi-sensor data fusion technology to achieve efficient collaboration between vision, lidar and IMU data, optimize the complementarity of each sensor, and reduce the limitations of a single sensor. Through the collaborative work of these sensors, accurate 2D plane maps can be quickly generated in complex environments, especially suitable for outdoor, between-building or sloped environments, overcoming the limitations of traditional 2D lidar.

[0034] 2. Low cost

[0035] When visual SLAM is running normally, the present invention first binds the radar frame and the visual frame in time, then compensates for the radar distortion, interpolates to obtain the posture at the corresponding time, and obtains the accurate posture according to scan-match, which is integrated into the 2D radar map to achieve fast and accurate construction of the 2D map without the need for complicated data fusion and posture matching processes, taking into account cost-effectiveness, mapping accuracy and mapping efficiency.

[0036] 3. Improve the mapping accuracy of 2D radar

[0037] The visual SLAM system provides a high-precision estimation of the camera's pose through image feature extraction and matching, and combines IMU data for dynamic compensation and optimization, so that the lidar can obtain more accurate initial pose information. In addition, through the scanning and matching of laser data, the pose can be further optimized to build a more accurate map. Due to the binding relationship between laser data and visual data, laser data uses the rich closed-loop information provided by vision to timely optimize all poses and maps.

[0038] 4. Map synchronization

[0039] While updating the 2D grid map, the present invention also builds a 3D visual map in real time. The two are carried out synchronously and coordinated through an optimization algorithm, eliminating the problem caused by the scale or coordinate system being out of sync between visual SLAM and laser SLAM. Since the same coordinate system is shared, the positioning information based on the 3D visual map can be directly used for navigation on the 2D map, improving the robot's autonomous navigation capability in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0043] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0044] Example 1

[0045] This embodiment provides a joint mapping method based on multi-sensor fusion, such as Figure 1 As shown, the method comprises the following steps:

[0046] S1, acquire image data, 2D lidar data and IMU (inertial measurement unit) data and perform data preprocessing.

[0047] The core of the present invention is a visual SLAM system based on a camera and an IMU. First, pre-processing operations such as de-distortion are performed on the image collected by the camera to ensure the accuracy of image quality and feature extraction.

[0048] In this embodiment, the preprocessing includes:

[0049] 1. Dedistortion: The camera lens may introduce distortion (such as radial distortion and tangential distortion). The image needs to be dedistorted using the intrinsic parameters and distortion coefficients obtained through camera calibration to ensure the accuracy of the image geometry.

[0050] 2. Image enhancement: Depending on the environmental conditions (such as insufficient lighting, noise, etc.), it may be necessary to perform image enhancement processing (such as histogram equalization, filtering, etc.) to improve the robustness of subsequent feature extraction.

[0051] S2, combined with IMU data, applies the SLAM algorithm of vision and IMU fusion to each visual frame of the image data to estimate the pose at each moment.

[0052] Specifically, S2 includes the following steps:

[0053] S21, feature extraction: extract feature points from the image through feature extraction algorithms (such as SIFT, SURF, ORB, etc.);

[0054] S22, feature matching: matching the extracted feature points with the existing map point cloud, and accurately estimating the relative motion of the camera between two frames in combination with IMU data;

[0055] S23, pose estimation: solve the camera pose based on the feature matching results.

[0056] With the continuous connection of the motion trajectories at each moment, the motion trajectory and spatial point cloud of the entire camera in the environment are eventually generated, thereby achieving high-precision positioning.

[0057] S3, temporally binds the visual frame and the laser frame in the 2D lidar data based on the timestamp.

[0058] Specifically, based on each point in the lidar frame, its absolute timestamp on the global time axis is determined according to its relative timestamp in the frame, and the time series is aligned based on the timestamp of the visual frame and the absolute timestamp of the lidar frame to complete the time binding.

[0059] S4, based on the IMU data, performs motion distortion correction on the 2D lidar data, and combines the pose estimation at each moment to perform temporal pose interpolation on the data of the current laser frame, and spatially binds the laser frame and the visual frame.

[0060] To ensure the consistency between the LiDAR data and the actual environment, this paper introduces the LiDAR data posture compensation and motion distortion correction technology, effectively solving the problem of the mismatch between the laser frame posture and the actual posture during the robot movement, and eliminating the LiDAR data distortion caused by the robot movement. Specifically, we use the posture estimation at each moment provided by the visual SLAM system to perform temporal posture interpolation on the data of the current laser frame.

[0061] In this embodiment, the posture interpolation is specifically as follows: for the translation part, linear interpolation is used to obtain an estimated value; for the rotation part, spherical linear interpolation is used to obtain an estimated value; and the estimated values ​​of the translation part and the rotation part are combined to obtain the final posture interpolation.

[0062] On this basis, the present invention makes full use of high-frequency IMU data to further optimize the pose estimation of LiDAR data. The acceleration and angular velocity data provided by the IMU can effectively compensate for the angular error in the LiDAR frame pose, especially in dynamic motion, through real-time angle compensation, to ensure that the pose of the laser frame is more accurate. At the same time, the motion distortion compensation is performed on each frame of LiDAR data through this IMU data. When the LiDAR data is optimized through these compensations, the environmental information provided will be more consistent with the spatial structure of the real scene, thereby providing a more accurate basis for subsequent 2D map updates.

[0063] In the present invention, the 2D laser radar is closely integrated with the visual SLAM system, and the planar observation information of the 2D laser radar is directly integrated into the visual SLAM. Specifically, the laser keyframe and the visual keyframe are bound in time and space. Once the posture information of the visual SLAM is optimized, the corresponding laser keyframe posture and the corresponding map will also be optimized synchronously.

[0064] In actual applications, the visual system may be affected by lighting changes, occlusion or other factors, causing the visual odometer to lose tracking. At this time, 2D LiDAR, as a supplementary sensor, can provide laser odometers in real time to help the system stably track the robot's posture in a short period of time until the visual system resumes normal operation. In addition, for the scale problem of visual SLAM, laser data can also be calibrated and aligned together.

[0065] The present invention overcomes the problem that 2D LiDAR cannot complete closed-loop detection in some complex scenes by integrating the advantages of visual SLAM and LiDAR. In the fusion scheme, visual data provides richer global information and can effectively perform closed-loop detection. When a closed loop is detected, all laser frame positions can be optimized in time, and the 2D map can be optimized, thereby improving the overall quality and accuracy of the map.

[0066] S5, using the laser scanning matching algorithm to perform local matching on the interpolated lidar data to construct a 2D map.

[0067] In the present invention, the posture of the current laser frame is interpolated and compensated based on the visual posture, and a relatively accurate posture is obtained for the interpolation update of the map data. However, in any case, there are still certain differences in the posture, and direct update will cause certain ghosting on the edge of the map. This embodiment adopts the laser scan matching (ScanMatch) algorithm commonly used in lasers, and matches the laser radar scanning data through an accurate local matching algorithm to obtain a more accurate posture to improve the accuracy and consistency of the map, and construct a more accurate and refined 2D plane map. ScanMatch is an algorithm for matching laser radar (LiDAR) data, mainly used for scan matching (Scan Matching) tasks in robot localization and mapping (SLAM). Its core goal is to estimate the relative motion of the robot (or sensor) by aligning two or more frames of laser radar scanning data, thereby achieving accurate estimation of the robot's posture.

[0068] While updating the 2D grid map, the 3D visual map is also constructed in real time. The two are carried out synchronously and coordinated through optimization algorithms, eliminating the problems caused by scale or coordinate system asynchrony between visual SLAM and laser SLAM. Since they share the same coordinate system, the positioning information based on the 3D visual map can be directly used for navigation on the 2D map, eliminating the problem of coordinate system asynchrony and improving the fusion accuracy and consistency of the map.

[0069] The main innovations of the above scheme are:

[0070] 1. Posture compensation and motion distortion correction of lidar data:

[0071] The pose estimation and IMU data provided by the visual SLAM system are used to perform pose interpolation compensation and motion distortion correction on the lidar data.

[0072] 2. Multi-sensor fusion mapping and positioning

[0073] The 2D radar, camera and IMU are integrated to build a complete VLSLAM (Visual-Laser-SLAM) system. Vision provides a more accurate posture for the 2D radar, and the 2D radar provides scale calibration for the vision and supplements the system when it is lost.

[0074] 3. Accurate 2D map construction and update:

[0075] When constructing a 2D map, the laser scan matching (ScanMatch) algorithm is used to accurately match the laser radar data locally to improve the accuracy and consistency of the 2D map. At the same time, the laser frame is bound to the visual frame, and the position and map of the laser frame can be optimized synchronously when the visual is optimized.

[0076] 4. Real-time construction of 3D visual maps and synchronous updates of 2D grid maps:

[0077] This solution builds 3D visual maps and 2D grid maps at the same time, and coordinates them through optimization algorithms to ensure that the two share the same coordinate system, eliminating the problem of coordinate system asynchrony. It ensures a high degree of consistency between visual SLAM and lidar SLAM, thereby improving map fusion accuracy and ensuring accuracy when navigating on 2D maps.

[0078] Example 2

[0079] This embodiment provides a joint mapping device based on multi-sensor fusion, including:

[0080] Data acquisition module: acquires image data, 2D lidar data and IMU data and performs data preprocessing;

[0081] Preliminary pose estimation module: Combined with IMU data, the SLAM algorithm combining vision and IMU fusion is applied to each visual frame of the image data to estimate the pose at each moment;

[0082] Time binding module: Temporally binds the visual frames and the laser frames in the 2D lidar data based on the timestamp;

[0083] Spatial binding module: It performs motion distortion correction on 2D lidar data based on IMU data, and interpolates the pose of the current laser frame in time based on the pose estimation at each moment, and spatially binds the laser frame and the visual frame.

[0084] Mapping module: The laser scanning matching algorithm is used to perform local matching on the interpolated lidar data to construct a 2D map.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0086] Example 3

[0087] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0088] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0089] The processing unit performs the various methods and processes described above, such as methods S1 to S5. For example, in some embodiments, methods S1 to S5 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S5 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S5 in any other appropriate manner (e.g., by means of firmware).

[0090] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0091] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A joint mapping method based on multi-sensor fusion, characterized in that: The method comprises the following steps: S1, acquire image data, 2D lidar data and IMU data and perform data preprocessing; S2, combined with IMU data, applies the SLAM algorithm of vision and IMU fusion to each visual frame of image data to estimate the pose at each moment; S3, temporally binds the visual frame and the laser frame in the 2D lidar data based on the timestamp; S4, based on the IMU data, performs motion distortion correction on the 2D lidar data, and combines the pose estimation at each moment to perform temporal pose interpolation on the data of the current laser frame, and spatially binds the laser frame and the visual frame; S5, using the laser scanning matching algorithm to perform local matching on the interpolated lidar data to construct a 2D map.

2. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: The S2 comprises the following steps: S21, feature extraction: extracting feature points from the image through a feature extraction algorithm; S22, feature matching: matching the extracted feature points with the existing map point cloud, and accurately estimating the relative motion of the camera between two frames in combination with IMU data; S23, pose estimation: solve the camera pose based on the feature matching results.

3. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: The S3 is specifically as follows: based on each point in the lidar frame, according to its relative timestamp in the frame, its absolute timestamp on the global time axis is determined, and based on the timestamp of the visual frame and the absolute timestamp of the lidar frame, the time series is aligned to complete the time binding.

4. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: The posture interpolation is specifically as follows: for the translation part, linear interpolation is used to obtain an estimated value, for the rotation part, spherical linear interpolation is used to obtain an estimated value, and the estimated values ​​of the translation part and the rotation part are combined to obtain the final posture interpolation.

5. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: In step S4, the acceleration and angular velocity data provided by the IMU data are used to compensate for the angular error in the LiDAR frame pose in real time, and the motion distortion compensation is performed on each frame of LiDAR data through the IMU data.

6. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: When the visual odometry loses tracking due to changes in lighting, occlusion or other factors and cannot obtain image data, a 2D lidar is used as a supplementary sensor to provide a laser odometry in real time. The 2D lidar data can be used to stably track the robot's position in a short period of time until the visual odometry resumes normal operation.

7. The joint mapping method based on multi-sensor fusion according to claim 1 is characterized in that: The method also includes: while updating the 2D grid map, constructing the 3D visual map in real time, the two are performed synchronously and coordinated through an optimization algorithm.

8. A joint mapping device based on multi-sensor fusion, characterized in that: include: Data acquisition module: acquires image data, 2D lidar data and IMU data and performs data preprocessing; Preliminary pose estimation module: Combined with IMU data, the SLAM algorithm combining vision and IMU fusion is applied to each visual frame of the image data to estimate the pose at each moment; Time binding module: Temporally binds the visual frames and the laser frames in the 2D lidar data based on the timestamp; Spatial binding module: It performs motion distortion correction on 2D lidar data based on IMU data, and interpolates the pose of the current laser frame in time based on the pose estimation at each moment, and spatially binds the laser frame and the visual frame. Mapping module: The laser scanning matching algorithm is used to perform local matching on the interpolated lidar data to construct a 2D map.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images