IGV multi-laser radar point cloud fusion method, device, equipment and medium

Abstract

The invention relates to the technical field of point cloud data, in particular to an IGV multi-laser radar point cloud fusion method, device and equipment and a medium, and the fusion method comprises the steps: obtaining original point cloud data obtained by a plurality of laser radars and inertial measurement data measured by an inertial measurement unit; wherein the timestamps of the plurality of laser radars and the inertial measurement unit are kept consistent; carrying out distortion removal on the original point cloud data based on the inertial measurement data to obtain first point cloud data; the laser radar coordinate system corresponding to each frame of first point cloud data is converted into an IGV body coordinate system, and second point cloud data is obtained after fusion; performing adaptive downsampling and miscellaneous point filtering on the second point cloud data to obtain third point cloud data; according to the scanning speed of the laser radar corresponding to each point cloud in the third point cloud data and the current scanning angle, calculating actual generation time; and sorting the third point cloud data according to the actual generation time to obtain fused point cloud data. According to the invention, more accurate and stable point cloud data can be obtained.

Description

Technical Field

[0001] This invention relates to the field of point cloud data technology, and in particular to a method, apparatus, equipment and medium for fusing point clouds from multiple IGV lidar systems. Background Technology

[0002] With the development of automated terminals, intelligent guided vehicles (IGVs) have become core transportation equipment, and the reliability of their autonomous navigation highly depends on accurate real-time perception of the surrounding environment. LiDAR, due to its high precision and immunity to lighting conditions, has become the mainstream sensor for IGV environmental perception. However, single LiDAR systems have inherent blind spots, especially in the area close to the vehicle and directly in front of or behind it, often obstructed by the vehicle body or containers.

[0003] To address this issue, multi-LiDAR fusion has become a viable solution. However, data from different LiDARs suffers from temporal asynchrony and independent coordinate systems. Directly fusing unprocessed point cloud data leads to motion distortion and temporal misalignment, reducing the accuracy of subsequent processing. Furthermore, a large amount of unordered fused point cloud data can also impact the real-time processing capabilities of the computing system. Therefore, there is an urgent need for a multi-LiDAR fusion system suitable for the IGV (Inlet Port Vehicle) working environment at a port, to achieve comprehensive environmental perception, ensure spatiotemporal synchronization at the data level, and output stable and accurately positioned environmental representation data. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, the present invention aims to provide a method, apparatus, device, and medium for fusing IGV multi-LiDAR point clouds, which can solve the problem that the positioning accuracy of point cloud data obtained by multi-LiDAR fusion is affected by motion distortion and time misalignment.

[0005] To solve the above problems, the present invention adopts the following technical solution:

[0006] According to a first aspect of the present invention, a method for fusing point clouds from multiple lidar sensors in an IGV is provided. The IGV is equipped with an inertial measurement unit and multiple lidar sensors, the total scanning range of which covers the perimeter and top of the IGV. The fusing method includes:

[0007] The system acquires raw point cloud data from multiple lidar sensors and inertial measurement data from an inertial measurement unit (IMU); the timestamps of the lidar sensors and the IMU are kept consistent.

[0008] The original point cloud data corresponding to each lidar is distorted based on inertial measurement data to obtain the first point cloud data.

[0009] The coordinate system of the LiDAR corresponding to the first point cloud data in each frame is converted into the vehicle coordinate system of the IGV, and all the converted first point cloud data are fused to obtain the second point cloud data.

[0010] Adaptive downsampling and noise filtering are applied to the second point cloud data to obtain the third point cloud data;

[0011] Based on the scanning speed and scanning angle of the LiDAR corresponding to each point cloud in the third point cloud data, the actual generation time of each point cloud is calculated; the third point cloud data is then reordered according to the actual generation time of each point cloud to obtain the fused point cloud data of multiple LiDARs.

[0012] In some embodiments, distortion correction processing is performed on the original point cloud data corresponding to each lidar based on inertial measurement data to obtain first point cloud data, including:

[0013] Inertial measurement data includes acceleration data and angular velocity data. The trajectory is calculated using the acceleration data and angular velocity data to obtain the IGV motion pose corresponding to the original point cloud data.

[0014] Based on the IGV motion pose, the motion deviation of the original point cloud data is adjusted to obtain the first point cloud data.

[0015] In some embodiments, converting the lidar coordinate system corresponding to the first point cloud data of each frame into the vehicle coordinate system of the IGV includes:

[0016] Calculate the rotation matrix and translation vector of each lidar coordinate system relative to the vehicle coordinate system;

[0017] The rotation matrix and translation vector are applied to the corresponding first point cloud data to complete the coordinate system transformation.

[0018] In some embodiments, adaptive downsampling of the second point cloud data includes: adaptive downsampling of the second point cloud data using a voxel grid filter algorithm.

[0019] In some embodiments, a voxel grid filter algorithm is used to divide the point cloud space corresponding to the second point cloud data into several voxels, and the voxel size of the corresponding region is adjusted according to the point cloud density of each region in the point cloud space. The voxel size is inversely proportional to the point cloud density.

[0020] In some embodiments, the actual generation time of each point cloud is calculated based on the scanning speed of the lidar corresponding to each point cloud in the third point cloud data and the scanning angle at that time, including:

[0021] Get the timestamp of each point cloud in the third point cloud data;

[0022] Calculate the time deviation for each point cloud based on the scanning speed of the lidar corresponding to each point cloud and the scanning angle at that time.

[0023] Based on the time deviation and timestamp of each point cloud, the actual generation time of each point cloud is obtained.

[0024] In some embodiments, methods for synchronizing the timestamps of multiple lidar units with those of an inertial measurement unit include: network time protocol, precision time protocol, or GPS signal synchronization.

[0025] A second aspect of the present invention provides a fusion device for multiple lidar point clouds of an IGV, wherein the IGV is equipped with an inertial measurement unit and multiple lidars, and the total scanning range of the multiple lidars covers the perimeter and top of the IGV. The fusion device includes:

[0026] The data acquisition module is used to acquire the raw point cloud data acquired by multiple lidars and the inertial measurement data measured by the inertial measurement unit; wherein the timestamps of the multiple lidars and the inertial measurement unit are consistent.

[0027] The distortion correction module is used to perform distortion correction on the original point cloud data corresponding to each lidar based on inertial measurement data to obtain the first point cloud data.

[0028] The coordinate system transformation module is used to convert the lidar coordinate system corresponding to the first point cloud data of each frame into the vehicle coordinate system of the IGV, and fuse all the transformed first point cloud data to obtain the second point cloud data.

[0029] The downsampling and filtering module is used to perform adaptive downsampling and noise filtering on the second point cloud data to obtain the third point cloud data.

[0030] The point cloud time sorting module is used to calculate the actual generation time of each point cloud based on the scanning speed and scanning angle of the LiDAR corresponding to each point cloud in the third point cloud data; and reorder the third point cloud data according to the actual generation time of each point cloud to obtain the fused point cloud data of multiple LiDARs.

[0031] A third aspect of the present invention provides an electronic device, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the fusion method of IGV multi-lidar point clouds as described above.

[0032] A fourth aspect of the present invention provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the fusion method of IGV multi-lidar point clouds as described above.

[0033] Due to the above technical solution, the present invention has the following beneficial effects:

[0034] This invention proposes a method, apparatus, device, and medium for fusing point clouds from multiple lidar sensors used in an IGV (Inertial Measurement Vehicle). The fusing method includes: acquiring raw point cloud data from multiple lidar sensors and inertial measurement data from an inertial measurement unit (IMU), ensuring that the timestamps of the multiple lidar sensors and the IMU are consistent to guarantee that the data collected by the multiple lidar sensors and the IMU are synchronized in time. Further, distortion correction is performed on the raw point cloud data based on the inertial measurement data to obtain first point cloud data. Since the inertial measurement data reflects the motion state of the IGV, eliminating motion distortion in the raw point cloud data using inertial measurement data yields more accurate first point cloud data. The lidar coordinate system corresponding to each frame of the first point cloud data is converted to the IGV vehicle coordinate system, and the resulting fused data is a second point cloud data, which includes more comprehensive and complete information about the surrounding environment. The second point cloud data is adaptively downsampled and filtered for noise to obtain the third point cloud data. Based on the scanning speed of the LiDAR corresponding to each point cloud in the third point cloud data and the original point cloud data, the actual generation time is calculated. The third point cloud data is sorted according to the actual generation time to obtain fused point cloud data. The fused point cloud data has more accurate real-time performance and can have better consistency and stability in different times and scenarios, thereby improving the accuracy of IGV positioning. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the present invention, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0036] Figure 1 This is a schematic diagram showing the positions of the inertial measurement unit and lidar on an IGV provided in one embodiment of the present invention;

[0037] Figure 2 This is a flowchart of an IGV multi-LiDAR point cloud fusion method provided in one embodiment of the present invention;

[0038] Figure 3 This is a flowchart of a method for distortion removal of raw point cloud data provided in an embodiment of the present invention;

[0039] Figure 4 This is a flowchart of a coordinate system transformation method provided in one embodiment of the present invention;

[0040] Figure 5This is a flowchart for calculating the actual point cloud generation time according to an embodiment of the present invention;

[0041] Figure 6 This is a structural diagram of an IGV multi-lidar point cloud fusion device provided in one embodiment of the present invention;

[0042] Figure 7 This is a block diagram of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] It should be noted that the terms "first," "first," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0045] In a port environment, Intelligent Guided Vehicles (IGVs) receive automatically generated operational instructions from the system to perform automated container transport tasks. The reliability of their autonomous navigation highly depends on accurate and real-time perception of the surrounding environment. Multiple lidar sensors can reduce the blind spots of IGVs; however, the disordered and complex point cloud data generated by multiple lidar sensors can affect the accuracy of IGVs' perception of their surroundings. Therefore, this invention proposes a method for fusing point clouds from multiple lidar sensors for IGVs, which can construct complete, accurate, and ordered point cloud data, thereby improving the stability and accuracy of IGV positioning.

[0046] The following describes, with reference to the accompanying drawings, a method for fusing IGV multi-LiDAR point clouds according to an embodiment of the present invention.

[0047] First, such as Figure 1The diagram illustrates the arrangement of the inertial measurement unit 1000 and the lidar 2000. The IGV to which the fusion method of this invention is applicable is equipped with an inertial measurement unit 1000 and multiple lidars 2000. The inertial measurement unit 1000 is located at the center of mass of the IGV and can measure the three-axis (X, Y, Z axes) acceleration data and three-axis angular velocity data of the IGV. Using the three-axis acceleration and angular velocity data, the attitude, position, and orientation changes of the IGV can be further calculated. The lidar 2000 includes two horizontal scanning lidars 2001 located at the right front and left rear of the IGV, respectively, and two upward scanning lidars 2002 located at the front and rear of the IGV. The two lidars 2001 jointly complete the horizontal scanning of the environment surrounding the IGV, while the two upward scanning lidars 2002 jointly complete the scanning of the area above the IGV. Since there are often high-altitude devices such as quay cranes in the high-altitude areas of the dock environment, the upward scanning lidars 2002 are needed to provide more comprehensive monitoring of the environment surrounding the IGV.

[0048] It should be noted that, Figure 1 The arrangement of the LiDAR 2000 is only an example and does not constitute a limitation on the embodiments of the present invention. The present invention may include more or fewer LiDAR 2000 than shown in the figure, as long as the total scanning range of multiple LiDAR 2000 can cover the four sides and above the IGV.

[0049] Specifically, such as Figure 2 As shown, the fusion method of IGV multi-LiDAR point clouds includes the following steps S100~S500:

[0050] Step S100: Acquire the raw point cloud data acquired by each of the multiple lidar units 2000 and the inertial measurement data measured by the inertial measurement unit 1000. The timestamps of the multiple lidar units and the inertial measurement unit are consistent.

[0051] Among them, the LiDAR 2000 adopts a surround-view LiDAR, which is a LiDAR system that achieves 360-degree all-round environmental perception through multi-beam laser scanning. It can build a high-precision three-dimensional point cloud map around the IGV in real time.

[0052] Multiple lidar units and inertial measurement units (IMUs) are synchronized by hardware trigger signals, such as a pulse signal sent by the main control system to activate multiple sensors simultaneously, ensuring consistent timestamps. Alternatively, they can be synchronized using a precision clock source, such as the PTP (Precise Time Protocol). Time synchronization ensures that different sensors collect data at the same point in time, allowing subsequent data processing and fusion to accurately reflect the actual situation, reducing errors caused by time differences, and improving data accuracy.

[0053] Step S200: Based on the inertial measurement data, perform distortion correction processing on the original point cloud data corresponding to each lidar to obtain the first point cloud data.

[0054] During the IGV's movement, each lidar 2000 scans around itself, generating raw point cloud data. Because the IGV is constantly moving, the position of each lidar 2000 changes continuously during the scanning process, resulting in motion distortion in the raw point cloud data. Inertial measurement data includes the IGV's acceleration and angular velocity data. By integrating the acceleration and angular velocity data, the real-time position and attitude changes of the IGV, i.e., the position and attitude changes of each lidar 2000, can be calculated. This is used to correct the motion distortion of the raw point cloud data corresponding to each lidar, obtaining the first point cloud data.

[0055] Step S300: Convert the lidar coordinate system corresponding to the first point cloud data of each frame into the vehicle coordinate system of the IGV, and fuse all the converted first point cloud data to obtain the second point cloud data.

[0056] Specifically, the spatial transformation relationship between the coordinate system of each LiDAR 2000 and the vehicle coordinate system of the IGV is obtained, namely the rotation matrix and translation vector. Using the rotation matrix and translation vector, the spatial transformation of the coordinate system of each LiDAR 2000 to the IGV's vehicle coordinate system is performed. This transforms the single-frame first point cloud data from multiple different coordinate systems into the same coordinate system, fusing them into a single-frame second point cloud data. The second point cloud data uses the IGV's vehicle coordinate system as its coordinate system. By uniformly mapping the first point cloud data of each LiDAR to the same IGV's vehicle coordinate system, the continuity of spatial information around the IGV and the accuracy of subsequent fusion are ensured.

[0057] Step S400: Perform adaptive downsampling and noise filtering on the second point cloud data to obtain the third point cloud data.

[0058] The adaptive downsampling of the second point cloud data involves adjusting the sampling density based on the local density of the second point cloud data. By assigning different sampling rates to different local point cloud data densities, the amount of point cloud data is reduced, which speeds up subsequent calculations. Adaptive downsampling also improves sampling reliability and preserves key information in the point cloud data.

[0059] Noise filtering can remove noise and irrelevant points from point cloud data, improving the overall quality of the data. These points are usually caused by sensor errors or external environmental influences, such as the IGV itself obstructing the point cloud data generated by the LiDAR 2000.

[0060] Step S500: Based on the scanning speed of the LiDAR 2000 corresponding to each point cloud in the third point cloud data and the scanning angle at that time, calculate the actual generation time of each point cloud; reorder the third point cloud data according to the actual generation time of each point cloud to obtain the fused point cloud data of multiple LiDARs.

[0061] Among them, each point cloud data in the third point cloud data contains the ID of its corresponding LiDAR 2000, as well as information such as the scanning angle and timestamp at that time, and can obtain information such as the scanning speed and scanning cycle of each LiDAR 2000.

[0062] Because the scanning cycle and scanning speed of each LiDAR 2000 differ, the timestamps collected by each LiDAR 2000 at the same moment will be different. Alternatively, at the same timestamp, the actual generation time of each point cloud in the raw point cloud data acquired by each LiDAR 2000 will be different. Therefore, calculating the actual generation time of each point cloud can resolve the temporal ambiguity problem in the third point cloud data, ensuring it is strictly ordered on the timeline.

[0063] This embodiment significantly improves the accuracy of point cloud data by time synchronization and motion distortion removal of the original point cloud data. By calculating spatial transformation relationships, the coordinate system of the first point cloud data generated by all LiDAR 2000s is set to the vehicle coordinate system of IGV, ensuring the continuity of spatial information reflected by the point cloud data. Adaptive downsampling and noise filtering improve data quality and reduce redundant data. Finally, the actual generation time of each point cloud is calculated, and the third point cloud data after the above processing is re-sorted by time, making the fusion of point cloud data generated by multiple LiDAR 2000s more consistent, avoiding information distortion caused by spatiotemporal asynchrony, and providing more accurate fused point cloud data for subsequent analysis and modeling.

[0064] In some embodiments, for step S200, distortion correction processing is performed on the original point cloud data corresponding to each lidar based on inertial measurement data to obtain the first point cloud data, such as... Figure 3 As shown, steps S201 to S202 are included:

[0065] Step S201: Inertial measurement data includes acceleration data and angular velocity data. The trajectory is calculated using the acceleration data and angular velocity data to obtain the IGV motion pose corresponding to the original point cloud data.

[0066] Specifically, by integrating the acceleration data over time, the displacement of the IGV relative to its initial position can be obtained, thus determining its current position coordinates. By integrating the angular velocity data over time, the rotation angles (pitch, roll, and yaw) of the IGV can be obtained, thereby determining the IGV's motion pose. The IGV's motion pose includes its three-dimensional position and orientation at a specific moment.

[0067] Step S202: Adjust the motion deviation of the original point cloud data according to the IGV motion pose to obtain the first point cloud data.

[0068] For example, step S201 uses inertial measurement data to calculate the sequence corresponding to the IGV motion pose. , express The translation vector of IGV at time t. express The rotation matrix of IGV at time t. For the t... A collection of raw point cloud data acquired by 2000 LiDAR units Each of the first and second digits can be calculated using a linear interpolation algorithm. The motion pose of a lidar 2000 at its acquisition time t , Indicates the first The translation vector of a LiDAR 2000 at its acquisition time t Indicates the first A 2000 lidar unit, at its acquisition time t, based on motion pose... Adjust the motion deviation of the original point cloud data.

[0069] In some embodiments, for step S300, the lidar coordinate system corresponding to the first point cloud data of each frame is converted into the vehicle coordinate system of the IGV, such as... Figure 4 As shown, steps S301 to S302 are included:

[0070] Step S301: Calculate the rotation matrix and translation vector of each lidar coordinate system relative to the vehicle coordinate system.

[0071] Step S302: Apply the rotation matrix and translation vector to the corresponding first point cloud data to complete the coordinate system transformation.

[0072] For example, for the first The first set of point cloud data collected by a lidar Arbitrary point cloud , Indicates the first The point at the th The coordinates in the lidar coordinate system are converted to the coordinates in the IGV vehicle body coordinate system using a coordinate transformation formula. The coordinate transformation formula is:

[0073] ;

[0074] in, Indicates the first The point cloud in the first Coordinates in a lidar coordinate system This is the rotation matrix used to transform the point cloud from the lidar coordinate system to the vehicle coordinate system. This is the translation vector that transforms the point from the lidar coordinate system to the vehicle coordinate system.

[0075] In some embodiments, step S400, adaptive downsampling of the second point cloud data, includes: adaptive downsampling of the second point cloud data using a voxel grid filter algorithm.

[0076] It should be noted that the use of a voxel grid filter algorithm to adaptively downsample the second point cloud data in this embodiment does not constitute a limitation on the adaptive downsampling method in this invention. Clustering algorithms or deep learning methods can also be used for adaptive downsampling, and this embodiment of the invention does not impose any restrictions on this.

[0077] Furthermore, a voxel grid filter algorithm is used to divide the point cloud space corresponding to the second point cloud data into several voxels. The voxel size of the corresponding region is adjusted according to the point cloud density of each region in the point cloud space. The voxel size is inversely proportional to the point cloud density.

[0078] In some embodiments, for step S500, based on the scanning speed of the LiDAR 2000 corresponding to each point cloud in the third point cloud data and the scanning angle at that time, the actual generation time of each point cloud is calculated, such as... Figure 5 As shown, steps S501 to S503 are included:

[0079] Step S501: Obtain the timestamp of each point cloud in the third point cloud data;

[0080] Step S502: Calculate the time deviation for each point cloud based on the scanning speed of the lidar corresponding to each point cloud and the scanning angle at that time;

[0081] Step S503: Based on the time deviation and timestamp time corresponding to each point cloud, obtain the actual generation time of each point cloud.

[0082] The following example illustrates this embodiment. Assume that a lidar 2000 corresponding to a certain point cloud in the third point cloud data rotates at a constant angular velocity ω. The scanning angle is The starting angle of the entire frame is Then its time deviation for:

[0083] .

[0084] Combined point cloud Recorded timestamps yield point clouds The actual generation time.

[0085] In some embodiments, methods for synchronizing the timestamps of multiple lidar units with those of an inertial measurement unit include: network time protocol, precision time protocol, or GPS signal synchronization.

[0086] In summary, the fusion method for multiple lidar point clouds of IGVs proposed in this invention provides a complete technical solution to the data fusion problem of multiple lidar point clouds of Intelligent Guided Vehicles (IGVs) in a dock environment. First, an inertial measurement unit (IMU) and multiple lidars are set up on the IGV to acquire the original point cloud data of each lidar and the inertial measurement data of the IMU, ensuring consistent timestamps. Then, distortion correction is performed on the original point cloud data based on the inertial measurement data to correct point cloud distortion caused by IGV motion, resulting in first point cloud data. Next, each frame of first point cloud data is transformed from the lidar coordinate system to the IGV vehicle coordinate system, and all data are fused to obtain second point cloud data. Adaptive downsampling and noise filtering are performed on the second point cloud data; the sampling rate is adjusted according to local density, and noise points are removed to reduce redundant data, resulting in third point cloud data. Finally, based on the scanning speed and scanning angle of the lidar corresponding to each point cloud in the third point cloud data, the actual generation time is calculated and reordered to obtain fused point cloud data, thus solving the temporal ambiguity problem. Ultimately, this provides IGV with more accurate and stable fused point cloud data, outputting stable environmental representations that can be used for precise positioning.

[0087] like Figure 6 As shown, a second aspect of the present invention provides a fusion device for multiple lidar point clouds of an IGV. The IGV is equipped with an inertial measurement unit and multiple lidars, the total scanning range of the multiple lidars covering the perimeter and top of the IGV. The fusion device 600 includes:

[0088] The data acquisition module 601 is used to acquire the raw point cloud data acquired by multiple lidars and the inertial measurement data measured by the inertial measurement unit; wherein the timestamps of the multiple lidars and the inertial measurement unit are consistent.

[0089] The distortion correction module 602 is used to perform distortion correction processing on the original point cloud data corresponding to each lidar based on inertial measurement data to obtain the first point cloud data.

[0090] The coordinate system transformation module 603 is used to convert the lidar coordinate system corresponding to the first point cloud data of each frame into the vehicle coordinate system of the IGV, and fuse all the transformed first point cloud data to obtain the second point cloud data.

[0091] The downsampling and filtering module 604 is used to perform adaptive downsampling and noise filtering on the second point cloud data to obtain the third point cloud data.

[0092] The point cloud time sorting module 605 is used to calculate the actual generation time of each point cloud based on the scanning speed and scanning angle of the LiDAR corresponding to each point cloud in the third point cloud data; and to re-sort the third point cloud data according to the actual generation time of each point cloud to obtain the fused point cloud data of multiple LiDARs.

[0093] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process can be found in the corresponding method embodiments, which will not be repeated here.

[0094] An embodiment of the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the IGV multi-LiDAR point cloud fusion method provided in the above embodiments.

[0095] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0096] Refer to the attached reference manual Figure 7The diagram shown is a block diagram of an electronic device 700 according to an embodiment of the present invention. The electronic device 700 may include one or more processors 702, system control logic 708 connected to at least one of the processors 702, system memory 704 connected to the system control logic 708, non-volatile memory (NVM) 706 connected to the system control logic 708, and network interface 710 connected to the system control logic 708.

[0097] Processor 702 may include one or more single-core or multi-core processors. Processor 702 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments herein, processor 702 may be configured to perform operations according to... Figures 2 to 5 One or more embodiments of the various embodiments shown.

[0098] In some embodiments, system control logic 708 may include any suitable interface controller to provide any suitable interface to at least one of the processors 702 and / or any suitable device or component communicating with system control logic 708.

[0099] In some embodiments, system control logic 708 may include one or more memory controllers to provide an interface to system memory 704. System memory 704 may be used to load and store data and / or instructions. In some embodiments, system memory 704 of electronic device 700 may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0100] NVM / Memory 706 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / Memory 706 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of HDD (Hard Disk Drive), CD (Compact Disc) drive, and DVD (Digital Versatile Disc) drive.

[0101] NVM / Storage 706 may include a portion of storage resources mounted on a device of Electronic Device 700, or it may be accessible by the device but is not necessarily part of the device. For example, NVM / Storage 706 may be accessed over a network via Network Interface 710.

[0102] Specifically, system memory 704 and NVM / memory 706 may each include a temporary copy and a permanent copy of instruction 720. Instruction 720 may include, when executed by at least one of processors 702, causing electronic device 700 to perform, as Figures 2 to 5 The instructions for the fusion method of IGV multi-LiDAR point clouds are shown. In some embodiments, the instructions 720, hardware, firmware and / or their software components may additionally / alternatively be placed in the system control logic 708, network interface 710 and / or processor 702.

[0103] Network interface 710 may include a transceiver for providing a radio interface to electronic device 700, thereby enabling communication with any other suitable device (such as a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 710 may be integrated into other components of electronic device 700. For example, network interface 710 may be integrated into at least one of the following: a communication module of processor 702, system memory 704, NVM / memory 706, and firmware device (not shown) with instructions, which, when at least one of processor 702 executes the instructions, enable electronic device 700 to implement... Figures 2 to 5 One or more embodiments of the various embodiments shown.

[0104] The network interface 710 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 710 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0105] In one embodiment, at least one of the processors 702 may be packaged together with the logic of one or more controllers for system control logic 708 to form a system package (SiP). In another embodiment, at least one of the processors 702 may be integrated on the same die with the logic of one or more controllers for system control logic 708 to form a system on chip (SoC).

[0106] The electronic device 700 may further include an input / output (I / O) device 712. The I / O device 712 may include a user interface enabling a user to interact with the electronic device 700; the peripheral component interface is designed to allow peripheral components to also interact with the electronic device 700. In some embodiments, the electronic device 700 may also include sensors for determining at least one type of environmental condition and location information related to the electronic device 700.

[0107] In some embodiments, the user interface may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash), and a keyboard.

[0108] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0109] In some embodiments, the sensor may include, but is not limited to, a gyroscope sensor, an accelerometer, a proximity sensor, an ambient light sensor, and a positioning unit. The positioning unit may also be part of or interact with the network interface 710 to communicate with components of the positioning network, such as Global Positioning System (GPS) satellites.

[0110] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 700. In other embodiments of the present invention, the electronic device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] An embodiment of the present invention also provides a computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing an IGV multi-LiDAR point cloud fusion method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the IGV multi-LiDAR point cloud fusion method provided in the above-described method embodiment.

[0112] Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0113] One embodiment of the present invention also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the IGV multi-LiDAR point cloud fusion method provided in the various optional implementations described above.

[0114] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0116] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0117] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fusing point clouds from multiple IGV lidar systems, characterized in that, The IGV is equipped with an inertial measurement unit and multiple lidar sensors. The total scanning range of the multiple lidar sensors covers the perimeter and top of the IGV. The fusion method includes: The system acquires raw point cloud data from each of the multiple lidars and inertial measurement data from the inertial measurement unit; wherein the timestamps of the multiple lidars and the inertial measurement unit are consistent. Based on the inertial measurement data, the original point cloud data corresponding to each lidar is subjected to distortion correction processing to obtain the first point cloud data; The coordinate system of the lidar corresponding to each frame of the first point cloud data is converted into the vehicle coordinate system of the IGV, and all the converted first point cloud data are fused to obtain the second point cloud data. The second point cloud data is subjected to adaptive downsampling and noise filtering to obtain the third point cloud data. Based on the scanning speed and scanning angle of the lidar corresponding to each point cloud in the third point cloud data, the actual generation time of each point cloud is calculated; the third point cloud data is then reordered according to the actual generation time of each point cloud to obtain fused point cloud data from multiple lidars.

2. The fusion method of IGV multi-LiDAR point clouds according to claim 1, characterized in that, Based on the inertial measurement data, distortion correction processing is performed on the original point cloud data corresponding to each lidar to obtain the first point cloud data, including: The inertial measurement data includes acceleration data and angular velocity data. The trajectory is calculated using the acceleration data and the angular velocity data to obtain the IGV motion pose corresponding to the original point cloud data. Based on the IGV motion pose, the motion deviation of the original point cloud data is adjusted to obtain the first point cloud data.

3. The fusion method of IGV multi-LiDAR point clouds according to claim 1, characterized in that, Converting the lidar coordinate system corresponding to the first point cloud data in each frame into the vehicle coordinate system of the IGV includes: Calculate the rotation matrix and translation vector of each of the lidar coordinate systems relative to the vehicle coordinate system; The rotation matrix and the translation vector are applied to the corresponding first point cloud data to complete the coordinate system transformation.

4. The fusion method of IGV multi-LiDAR point clouds according to claim 1, characterized in that, Adaptive downsampling of the second point cloud data includes: using a voxel grid filter algorithm to adaptively downsample the second point cloud data.

5. The fusion method of IGV multi-LiDAR point clouds according to claim 4, characterized in that, The point cloud space corresponding to the second point cloud data is divided into several voxels using a voxel grid filter algorithm. The voxel size of the corresponding region is adjusted according to the point cloud density of each region in the point cloud space. The voxel size is inversely proportional to the point cloud density.

6. The fusion method of IGV multi-LiDAR point clouds according to claim 1, characterized in that, Based on the scanning speed and scanning angle of the lidar corresponding to each point cloud in the third point cloud data, the actual generation time of each point cloud is calculated, including: Obtain the timestamp of each point cloud in the third point cloud data; The time deviation for each point cloud is calculated based on the scanning speed of the lidar corresponding to each point cloud and the scanning angle at that time. Based on the time deviation and timestamp time corresponding to each point cloud, the actual generation time of each point cloud is obtained.

7. The fusion method of IGV multi-LiDAR point clouds according to claim 1, characterized in that, Methods for keeping the timestamps of the multiple lidars consistent with those of the inertial measurement unit include: network time protocol, precision time protocol, or GPS signal synchronization.

8. A fusion device for IGV multi-lidar point clouds, characterized in that, The IGV is equipped with an inertial measurement unit and multiple lidar sensors. The total scanning range of the multiple lidar sensors covers the perimeter and top of the IGV. The fusion device includes: The data acquisition module is used to acquire the raw point cloud data acquired by each of the multiple lidars and the inertial measurement data measured by the inertial measurement unit; wherein the timestamps of the multiple lidars and the inertial measurement unit are consistent. The distortion correction module is used to perform distortion correction processing on the original point cloud data corresponding to each lidar based on the inertial measurement data to obtain the first point cloud data. The coordinate system transformation module is used to convert the lidar coordinate system corresponding to each frame of the first point cloud data into the vehicle coordinate system of the IGV, and fuse all the transformed first point cloud data to obtain the second point cloud data. The downsampling and filtering module is used to perform adaptive downsampling and noise filtering on the second point cloud data to obtain the third point cloud data. The point cloud time sorting module is used to calculate the actual generation time of each point cloud based on the scanning speed and scanning angle of the lidar corresponding to each point cloud in the third point cloud data; and to reorder the third point cloud data according to the actual generation time of each point cloud to obtain fused point cloud data of multiple lidars.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the IGV multi-lidar point cloud fusion method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the IGV multi-lidar point cloud fusion method as described in any one of claims 1-7.

Images