Multi-sensor information fusion method and device

Abstract

The application provides a multi-sensor information fusion method and device, and relates to the field of artificial intelligence. The method comprises 2D image features; determining initial object information, wherein the initial object information comprises position information of an initial object and 3D point cloud features of the initial object; determining a plurality of 2D image features corresponding to each 3D point cloud feature of the initial object through a trained multi-sensor information fusion model to obtain fusion information. In the multi-sensor information fusion, the application can fully utilize high-resolution color image information and improve the robustness of multi-sensor information fusion, thereby improving the detection performance and robustness of a three-dimensional target detector.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method and apparatus for multi-sensor information fusion. Background Technology

[0002] With societal development, intelligent transportation equipment, smart home devices, robots, and other intelligent terminals are gradually entering people's daily lives. Sensors play a crucial role in these intelligent terminals. Various sensors installed on intelligent terminals, such as millimeter-wave radar, lidar, cameras, and ultrasonic radar, perceive the surrounding environment, collect data, identify and track moving objects, and recognize stationary scenes such as lane lines and signs. They also combine this data with navigation and map data for path planning. Among these, lidar and cameras are the two most important sensors in 3D target detection. LiDAR provides accurate depth information, while cameras capture high-resolution color images rich in semantic information.

[0003] Currently, with the development of deep learning technology and the technological innovation of LiDAR, the detection performance of 3D target detection based on 3D point clouds using deep learning-driven data-driven methods far surpasses that of traditional devices. However, the 3D point clouds acquired by LiDAR are sparse, limiting its ability to detect small or distant objects. Therefore, improving the robustness of multi-sensor information fusion and the utilization rate of high-resolution color image information to enhance the detection performance of 3D target detectors is a pressing issue. Summary of the Invention

[0004] This application provides a multi-sensor information fusion method and apparatus, which can make full use of high-resolution color image information and improve the robustness of multi-sensor information fusion, thereby improving the detection performance and robustness of the three-dimensional target detector.

[0005] Firstly, a multi-sensor information fusion method is provided, which includes: acquiring 2D image features; determining initial object information, wherein the initial object information includes the position information of the initial object and the 3D point cloud features of the initial object; and determining multiple 2D image features corresponding to each 3D point cloud feature of the initial object based on the initial object information and the 2D image features, through a trained multi-sensor information fusion model, to obtain fused information.

[0006] It should be understood that 2D image features can be the output result of inputting 2D image information captured by a camera device into a 2D feature extraction model.

[0007] As one possible implementation, the initial object information can be determined by random determination or by querying the initialization process.

[0008] In this embodiment, the pre-trained multi-sensor information fusion model enables the correlation between point cloud features and image features to transcend the traditional one-to-one "hard correlation" between point clouds and pixels generated by a calibration matrix in point-by-point fusion. For example, one point cloud feature can correspond to multiple image features. Therefore, the pre-trained multi-sensor information fusion model in this embodiment makes the correlation between point cloud features and image features more flexible, effectively mitigating the problems of insufficient utilization of high-resolution image information in multi-sensor information fusion caused by this "hard correlation" and the low robustness of multi-sensor information fusion caused by using damaged image features. This improves the robustness of multi-sensor information fusion and the utilization rate of high-resolution images, thereby enhancing the accuracy of subsequent 3D target detection and recognition, as well as target tracking.

[0009] In some possible implementations, the trained multi-sensor information fusion model includes a cross-attention mechanism.

[0010] In the embodiments of this application, the cross-attention mechanism in the trained multi-sensor fusion model can effectively create a flexible correlation between point cloud features and image features, thereby improving the robustness of multi-sensor information fusion and the utilization rate of high-resolution images, thus improving the accuracy of subsequent 3D target detection and recognition as well as the accuracy of target tracking.

[0011] In some possible implementations, the target 3D point cloud features are obtained; the target 3D point cloud features are queried and initialized to determine the initial object information, wherein the position information of the initial object is the center position information of the initial object, and the 3D point cloud features of the initial object are the target 3D point cloud features of the initial object.

[0012] It should be understood that the target 3D point cloud features can be initial 3D point cloud features or 3D point cloud features fused with 2D image features.

[0013] In this embodiment, by querying the initialization process, the initial object information used for subsequent 3D object detection is made closer to the center position of the object. Compared with the traditional input-independent initialization method, it is not necessary to go through multiple layers of networks to continuously update the position of the initial object information. Therefore, obtaining fused information based on input-related initial object information can greatly improve the convergence speed and running efficiency of subsequent 3D object detection, improve algorithm efficiency, and reduce network parameters.

[0014] In some possible implementations, the initialization operation of querying the target 3D point cloud features to determine the initial object information includes: using the target 3D point cloud features as input to the initial object classification model to obtain M candidate initial object information, which includes the category probability value of the candidate initial object; and determining N of the M candidate initial object information as the initial object information based on the category probability values ​​of the M candidate initial objects, where N is a positive integer less than M and M is a positive integer.

[0015] In some possible implementations, a trained multi-sensor information fusion model is used to determine multiple 2D image features corresponding to each 3D point cloud feature of the initial object to obtain fusion information. This includes: using the initial object information and the target 3D point cloud features as input to the target object information determination model to obtain target object information, which includes the global target 3D point cloud features of the initial object; and using the trained multi-sensor information fusion model to determine multiple 2D image features corresponding to each global target 3D point cloud feature of the initial object to obtain fusion information.

[0016] In this embodiment, the cross-attention mechanism of the second multi-head attention allows for a flexible point-to-point association between target object information, including global 3D point cloud features, and 2D image features. This ensures that the fusion of image features and point cloud features is unaffected by the number of laser points, fully utilizing high-resolution image information. For example, when the number of laser point clouds is sparse, the 3D point cloud features and 2D image features can have a one-to-many association, thereby fully utilizing high-resolution image information. Furthermore, the target object information including global 3D point cloud features enriches the fusion information of the trained multi-sensor information fusion model, leading to more accurate determination of subsequent target detection boxes.

[0017] In some possible implementations, a trained multi-sensor information fusion model is used to determine multiple 2D image features corresponding to each global target 3D point cloud feature of the initial object to obtain fusion information. This includes: using the target object information as input to the initial detection box model to obtain the initial detection box; and using the trained multi-sensor information fusion model to determine multiple 2D image features corresponding to each global target 3D point cloud feature of the initial object to obtain fusion information. The spatially constrained cross-attention mechanism in the trained multi-sensor information fusion model is a cross-attention mechanism constrained by the initial detection box. The spatially constrained cross-attention mechanism is used for multi-sensor information fusion after spatially constraining the 2D image features.

[0018] In this embodiment, by using a spatially defined cross-attention mechanism that considers both spatial relevance and content relevance, image features and point cloud features are fused. This effectively filters out relevant image features from the image features and removes irrelevant or damaged image features, thereby improving the robustness of multi-sensor information fusion and thus enhancing the robustness of the 3D detector. In addition, reducing the influence of irrelevant image features during the fusion process can accelerate the convergence speed of the trained multi-sensor information fusion model.

[0019] In some possible implementations, obtaining target 3D point cloud features includes: obtaining initial 3D point cloud features; using 2D image features and initial 3D point cloud features as input to an initial object information fusion model to obtain target 3D point cloud features, wherein the target 3D point cloud features are 3D point cloud features fused with 2D image features.

[0020] In this embodiment of the application, the fused 3D point cloud features include not only 3D point cloud features but also image features, thereby avoiding the limitation of the subsequent determination of the target detection box being restricted by the natural sparsity of the laser point cloud.

[0021] In some possible implementations, 2D image features are defined as 2D image features that have undergone vertical compression.

[0022] It should be noted that the 2D image features after vertical compression can be understood as the 2D image features compressed along the height dimension.

[0023] In this embodiment of the application, the 2D image features are compressed along the vertical direction to compress the 2D image features, which greatly saves video memory overhead without losing important information.

[0024] In some possible implementations, the target 3D point cloud features are target bird's-eye view BEV features.

[0025] Secondly, a multi-sensor information fusion device is provided. The device includes an acquisition unit, a determination unit, and a fusion unit. The acquisition unit is used to acquire 2D image features; the determination unit is used to determine initial object information, wherein the initial object information includes the position information of the initial object and the 3D point cloud features of the initial object; and the fusion unit is used to determine multiple 2D image features corresponding to each 3D point cloud feature of the initial object through a trained multi-sensor information fusion model to obtain fused information.

[0026] It should be understood that 2D image features can be the output result of inputting 2D image information captured by a camera device into a 2D feature extraction model.

[0027] As one possible implementation, the initial object information can be determined by random determination or by querying the initialization process.

[0028] In this embodiment, the pre-trained multi-sensor information fusion model enables the correlation between point cloud features and image features to go beyond the traditional one-to-one "hard correlation" between point clouds and pixels in point-to-pixel fusion. For example, one point cloud feature can correspond to multiple image features. Therefore, the pre-trained multi-sensor information fusion model in this embodiment makes the correlation between point cloud features and image features more flexible, effectively alleviating the problem of insufficient utilization of high-resolution image information in multi-sensor information fusion caused by this "hard correlation" and the problem of low robustness of multi-sensor information fusion caused by using damaged image features. This improves the robustness of multi-sensor information fusion and the utilization rate of high-resolution images, thereby improving the accuracy of subsequent 3D target detection and recognition, as well as the accuracy of target tracking.

[0029] In some possible implementations, the trained multi-sensor information fusion model includes a cross-attention mechanism.

[0030] In the embodiments of this application, the cross-attention mechanism in the trained multi-sensor fusion model can effectively create a flexible correlation between point cloud features and image features, thereby improving the robustness of multi-sensor information fusion and the utilization rate of high-resolution images, thus improving the accuracy of subsequent 3D target detection and recognition as well as the accuracy of target tracking.

[0031] In some possible implementations, the acquisition unit is used to acquire the target 3D point cloud features; the determination unit is used to perform a query initialization operation on the target 3D point cloud features to determine the initial object information, wherein the position information of the initial object is the center position information of the initial object, and the 3D point cloud features of the initial object are the target 3D point cloud features of the initial object.

[0032] It should be understood that the target 3D point cloud features can be initial 3D point cloud features or 3D point cloud features fused with 2D image features.

[0033] In this embodiment, by querying the initialization process, the initial object information used for subsequent 3D object detection is made closer to the center position of the object. Compared with the traditional input-independent initialization method, it is not necessary to go through multiple layers of networks to continuously update the position of the initial object information. Therefore, obtaining fused information based on input-related initial object information can greatly improve the convergence speed and running efficiency of subsequent 3D object detection, improve algorithm efficiency, and reduce network parameters.

[0034] In some possible implementations, the determining unit is specifically used to: take the target 3D point cloud features as input to the initial object classification model to obtain M candidate initial object information, the candidate initial object information including the class probability value of the candidate initial object; and determine N of the M candidate initial object information as the initial object information based on the class probability value of the M candidate initial objects, where N is a positive integer less than M and M is a positive integer.

[0035] In some possible implementations, the fusion unit is specifically used to: take the initial object information and the target 3D point cloud features as input to the target object information determination model to obtain the target object information, which includes the global target 3D point cloud features of the initial object; and determine multiple 2D image features corresponding to each of the global target 3D point cloud features of the initial object through the trained multi-sensor information fusion model to obtain fused information.

[0036] In this embodiment, the cross-attention mechanism of the second multi-head attention allows for a flexible point-to-point association between target object information, including global 3D point cloud features, and 2D image features. This ensures that the fusion of image features and point cloud features is unaffected by the number of laser points, fully utilizing high-resolution image information. For example, when the number of laser point clouds is sparse, the 3D point cloud features and 2D image features can have a one-to-many association, thereby fully utilizing high-resolution image information. Furthermore, the target object information including global 3D point cloud features enriches the fusion information of the trained multi-sensor information fusion model, leading to more accurate determination of subsequent target detection boxes.

[0037] In some possible implementations, the fusion unit is specifically used to: take the target object information as input to the initial detection box model to obtain the initial detection box; and determine multiple 2D image features corresponding to each of the global target 3D point cloud features of the initial object through the trained multi-sensor information fusion model to obtain fusion information. The spatially constrained cross-attention mechanism in the trained multi-sensor information fusion model is a cross-attention mechanism constrained by the initial detection box, and the spatially constrained cross-attention mechanism is used for multi-sensor information fusion after spatially constraining the 2D image features.

[0038] In this embodiment, by using a spatially defined cross-attention mechanism that considers both spatial relevance and content relevance, image features and point cloud features are fused. This effectively filters out relevant image features from the image features and removes irrelevant or damaged image features, thereby improving the robustness of multi-sensor information fusion and thus enhancing the robustness of the 3D detector.

[0039] In some possible implementations, the acquisition unit is specifically used to: acquire initial 3D point cloud features; and use 2D image features and initial 3D point cloud features as input to an initial object information fusion model to obtain target 3D point cloud features, wherein the target 3D point cloud features are 3D point cloud features fused with 2D image features.

[0040] In this embodiment of the application, the fused 3D point cloud features include not only 3D point cloud features but also image features, thereby avoiding the limitation of the subsequent determination of the target detection box being restricted by the natural sparsity of the laser point cloud.

[0041] In some possible implementations, 2D image features are defined as 2D image features that have undergone vertical compression.

[0042] It should be noted that the 2D image features after vertical compression can be understood as the 2D image features compressed along the height dimension.

[0043] In this embodiment of the application, the 2D image features are compressed along the vertical direction to compress the 2D image features, which greatly saves video memory overhead without losing important information.

[0044] In some possible implementations, the target 3D point cloud features are target bird's-eye view BEV features.

[0045] Thirdly, an apparatus is provided, comprising a unit for performing a multi-sensor information fusion method according to any of the implementations of the first aspect described above.

[0046] Fourthly, an apparatus is provided, comprising a processing unit and a storage unit, wherein the storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the apparatus to perform any of the possible methods in the first aspect.

[0047] Optionally, the processing unit may be a processor, and the storage unit may be a memory, wherein the memory may be an on-chip storage unit (e.g., registers, caches, etc.) or an off-chip storage unit within a smart device (e.g., read-only memory, random access memory, etc.).

[0048] Fifthly, a smart device is provided, comprising the multi-sensor information fusion device described in the second aspect, the device described in the third aspect, or the device described in the fourth aspect.

[0049] In some possible implementations, the smart device may include vehicles, etc.

[0050] In a sixth aspect, a system is provided, comprising a plurality of sensors and a multi-sensor information fusion device, wherein the multi-sensor information fusion device may be any one of the multi-sensor information fusion devices described in the second aspect above.

[0051] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the multi-sensor information fusion device may be located in a cloud server.

[0052] In conjunction with the sixth aspect, in some implementations of the sixth aspect, the system also includes means for receiving instructions sent by a cloud server.

[0053] Eighthly, a computer-readable medium is provided that stores program code, which, when run on a computer, causes the computer to perform the method described in the first aspect. Attached Figure Description

[0054] Figure 1 This is a functional block diagram of the vehicle 100 provided in an embodiment of this application;

[0055] Figure 2 This is a schematic diagram of sensor distribution applied to a vehicle 100 provided in an embodiment of this application;

[0056] Figure 3 This is a schematic block diagram of the system architecture provided in the embodiments of this application;

[0057] Figure 4 This is a schematic block diagram of another system architecture provided in an embodiment of this application;

[0058] Figure 5 This is a flowchart illustrating a multi-sensor information fusion method provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram illustrating feature extraction of 3D point cloud information and 2D image information provided in an embodiment of this application;

[0060] Figure 7 This is a schematic diagram of an initial object information determination method provided in an embodiment of this application;

[0061] Figure 8 This is a schematic diagram of a multi-sensor information fusion method provided in an embodiment of this application;

[0062] Figure 9 This is a schematic diagram of the structure of a trained multi-sensor information fusion model provided in an embodiment of this application;

[0063] Figure 10 This is a schematic diagram of another multi-sensor information fusion method provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of a method for determining target object information provided in an embodiment of this application;

[0065] Figure 12 This is a schematic diagram of the structure of a target object information determination model provided in an embodiment of this application;

[0066] Figure 13 This is a spatially confined visualization diagram of cross-attention provided in an embodiment of this application;

[0067] Figure 14 This is a schematic diagram of a method for enhancing a target 3D point cloud provided in an embodiment of this application;

[0068] Figure 15 This is a schematic diagram of a target detection box acquisition method provided in an embodiment of this application;

[0069] Figure 16 This is a schematic diagram of a multi-sensor information fusion device provided in an embodiment of this application;

[0070] Figure 17 This is a schematic diagram of the hardware structure of a multi-sensor information fusion device provided in this application.

[0071] Figure 18 A schematic diagram of a system 1800 provided in an embodiment of this application is shown. Detailed Implementation

[0072] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0073] Figure 1 This is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 120, a display device 130, and a computing platform 150. The sensing system 120 may include several sensors for sensing information about the environment surrounding the vehicle 100. For example, the sensing system 120 may include a positioning system, which may be a Global Positioning System (GPS), or a BeiDou system or other positioning systems, an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and one or more of a camera device.

[0074] Some or all of the functions of vehicle 100 can be controlled by computing platform 150. Computing platform 150 may include processors 151 to 15n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. In addition, it can also be hardware circuitry designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. Furthermore, the computing platform 150 may also include a memory for storing instructions. Some or all of the processors 151 to 15n can call the instructions in the memory to execute them and achieve the corresponding functions.

[0075] Vehicle 100 may include an advanced driving assistance system (ADAS). ADAS utilizes various sensors on the vehicle (including but not limited to: lidar, millimeter-wave radar, camera devices, ultrasonic sensors, global positioning system, inertial measurement unit) to acquire information from the vehicle's surroundings, and analyzes and processes the acquired information to achieve functions such as obstacle perception, target recognition, vehicle positioning, path planning, and driver monitoring / alerts, thereby improving the safety, automation, and comfort of driving the vehicle.

[0076] In this application, "vehicle" can include one or more different types of means of transport, or one or more different types of transport vehicles or movable objects that operate or move on land (e.g., highways, roads, railways, etc.), water (e.g., waterways, rivers, oceans, etc.), or in space. For example, a vehicle can include a car, bicycle, motorcycle, train, subway, airplane, ship, aircraft, robot, or other types of transport vehicles or movable objects.

[0077] Figure 2 This is a schematic diagram of sensor distribution applied to a vehicle 100 provided in an embodiment of this application. It should be understood that... Figure 2 This is merely an illustrative diagram illustrating one possible sensor distribution method; other distribution methods are possible, and this application does not limit the scope of such methods. Figure 2 As shown, the sensors distributed on the vehicle 100 include a millimeter-wave radar 201, a camera device 202, and a lidar 203, and may also include... Figure 2 Other sensors not shown are not limited in this application embodiment. For example, the maximum sensing distance of the lidar is about 150 meters, the maximum sensing distance of the camera device is about 200 meters, the maximum sensing distance of the long-range millimeter-wave radar is about 250 meters, and the maximum sensing distance of the medium / short-range millimeter-wave radar is about 120 meters.

[0078] Figure 3 This is a schematic block diagram of the system architecture provided in the embodiments of this application. Figure 3 As shown, the system includes sensors and a multi-sensor information fusion device, which can also be called a multi-sensor information fusion unit. For example, the sensors may include... Figure 1 One or more of the sensors in the sensing system 120 can be configured according to... Figure 2 The distribution diagram shown is based on vehicle 100, but it can also be deployed in other ways. The multi-sensor information fusion device can be deployed on... Figure 1 The detection can be performed on the on-board computing platform 150, or it can be deployed on a cloud server and detected through the cloud server, and the fused information can be transmitted to the computing platform 150.

[0079] Sensors are used to perceive the vehicle and its surrounding environment and acquire data. For example, sensors may include one or more cameras and one or more LiDAR sensors. The data output by the sensors may be video stream data acquired by the cameras and point cloud information acquired by the LiDAR sensors.

[0080] Multi-sensor information fusion devices can be used to acquire data from multiple sensors and perform feature fusion on the data from multiple sensors to obtain fused information.

[0081] In one embodiment, the multi-sensor information fusion device can be deployed inside the vehicle as a hardware module and / or software module, for example, on a computing platform 150, or the sensor detection device can be a computing platform located in a cloud server. For example, Figure 4 A schematic block diagram of another system architecture provided in an embodiment of this application is shown. This system architecture includes sensors on a vehicle and a cloud server. The vehicle can transmit data collected by the sensors to the cloud server via a network. A multi-sensor information fusion device in the cloud server can perform feature fusion on the data collected by the sensors to obtain fused information. The cloud server can transmit the fused information to the vehicle via the network, for example, to a corresponding processor in the vehicle's computing platform 150.

[0082] To facilitate understanding of the technical solution of this application, a brief introduction to the concepts involved in this application will be given first.

[0083] (1) Attention mechanism (AM)

[0084] In computer vision systems, attention mechanisms simulate human attention, enabling neural networks to focus on a subset of important input information. In other words, attention mechanisms can be viewed as a dynamic selection process, adaptively weighting features based on the relevance or importance of the input. This application involves two attention mechanisms: self-attention and cross-attention.

[0085] Self-attention mechanisms, also known as internal attention mechanisms, reduce reliance on external information and excel at capturing the internal correlations of features. In other words, self-attention mechanisms aim to model the internal correlations of the same set of input features to enhance their own features and obtain more informative feature representations.

[0086] Cross-attention mechanisms are generally used to construct cross-correlation between two different sets of input features, obtain the relationship between the two sets of input features, and achieve the effect of fusing information onto one set of input features.

[0087] (2) Transformer model

[0088] The Transformer model is a deep learning model that includes self-attention and cross-attention mechanisms, both of which primarily take the form of multi-head attention. Multi-head attention allows the Transformer model to focus on information from different representation subspaces at different locations.

[0089] (3) Convolutional Neural Network (CNN)

[0090] A convolutional neural network (CNN) is a type of neural network with convolutional kernels. A convolutional kernel can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. A convolutional layer typically contains several feature planes, each of which can be composed of rectangularly arranged neural units.

[0091] Currently, there are various methods for multi-sensor information fusion. One approach is to understand data collected by different sensors as different modal information. For example, multi-sensor information fusion can be the fusion of 2D image information acquired by a camera and 3D point cloud information acquired by a lidar system. Currently, the fusion of 2D image information and 3D point cloud information is commonly used to enhance the performance of 3D target detection devices. For multi-sensor information involving 2D image information and 3D point cloud information, specific fusion methods can be categorized into coarse-grained fusion and fine-grained fusion. Coarse-grained fusion can be further divided into result-level fusion and candidate box-level fusion.

[0092] Result-level fusion involves performing target detection on 2D images and 3D point clouds separately, and then fusing the detection results from both with multi-sensor information. In this device, the target detection devices for 2D images and 3D point clouds are independent of each other, and the 2D image information and 3D point cloud information are not fused during the detection process. Therefore, result-level fusion does not fully utilize the complementarity between 2D images and 3D images to improve the detection performance of the 3D target detection device.

[0093] Candidate box-level fusion involves extracting 2D image features and 3D point cloud features from initial candidate boxes using a neural network, and then using these two features to obtain the final target detection box. Although candidate box-level fusion integrates 2D image features and 3D point cloud features during target detection, the fusion process is coarse-grained. Therefore, the performance improvement of a 3D target detection device using this candidate box-level fusion compared to a 3D target detection device that only uses 3D point cloud information is not significant.

[0094] One fine-grained fusion method is point-by-point fusion. First, a one-to-one correspondence between the pixels of the 3D point cloud and the 2D image is determined through the calibration matrix between the camera device and the lidar. Then, based on this one-to-one correspondence, visual information is obtained from the 2D image information to enhance the information of the 3D point cloud, thereby providing richer semantic information for the 3D point cloud which only has geometric features. Before extracting features through the neural network, point-by-point fine-grained fusion is performed on the 2D image and the 3D point cloud. The 3D target detection device using this fusion method has better detection performance.

[0095] However, point-by-point fusion has three main problems. First, because the relationship between 3D point clouds and 2D image pixels in point-by-point fusion is one-to-one, the 2D image information that can serve as supplementary information to the 3D point cloud depends on the number of 3D point clouds. When a LiDAR collects data from small or distant objects, the number of 3D point clouds acquired is limited. Therefore, when performing point-by-point fusion on the 3D point clouds of small or distant objects, only the information corresponding to the pixels in the 2D image of the 3D point cloud can be utilized. Thus, even if the 2D image information of small or distant objects contains rich semantic information, the high-resolution 2D image information cannot be fully utilized. Second, the process of establishing the one-to-one correspondence between 3D point clouds and 2D image pixels in point-by-point fusion is highly dependent on the calibration matrix between the camera device and the LiDAR. Therefore, the accuracy requirement for this calibration matrix is ​​relatively high. An erroneous calibration matrix will affect the results of multi-sensor information fusion, thereby affecting the detection performance of the 3D target detection device. Finally, if the 2D image captured by the camera device is damaged due to environmental factors, such as poor ambient lighting or excessive lighting, the 2D image information will be damaged. Since the process of establishing the one-to-one correspondence between the 3D point cloud and the 2D image pixels in the point-by-point fusion does not take into account the actual situation of the 2D image information, the damaged 2D image information will also be fused with the 3D point cloud information, thereby affecting the detection performance of the 3D target detection device.

[0096] To address the aforementioned problems, this application proposes a multi-sensor fusion method and apparatus, which will be described below in conjunction with... Figures 5 to 15 illustrate.

[0097] Figure 5 This is a flowchart illustrating a multi-sensor information fusion method provided in an embodiment of this application.

[0098] S501, acquire 2D image features.

[0099] For example, 2D image features can be the output result of inputting 2D image information captured by the camera device into a 2D feature extraction model. The embodiments of this application do not limit the 2D feature extraction model.

[0100] S502, determine the initial object information, wherein the initial object information includes the position information of the initial object and the 3D point cloud features of the initial object.

[0101] As one possible implementation, the initial object information can be determined by random determination or by querying the initialization process.

[0102] It should be understood that the position of the initial object information obtained through a random determination method is the initial object random position.

[0103] It should be understood that the initial object's position information obtained through query initialization is the center position information of the initial object, which is closer to the center of the initial object.

[0104] S503 uses a trained multi-sensor information fusion model to determine multiple 2D image features corresponding to each 3D point cloud feature of the initial object in order to obtain fused information.

[0105] It should be understood that the initial object information includes multiple 3D point cloud features of the initial object.

[0106] It should be noted that the trained multi-sensor information fusion model can be any neural network model that achieves multi-sensor information fusion by realizing a flexible correlation between 3D point cloud features and 2D image features; this application embodiment does not impose any limitations on this. Furthermore, the flexible correlation can be a one-to-many correlation between 3D point cloud features and 2D image features.

[0107] For example, the trained multi-sensor information fusion model can be a neural network model including a cross-attention mechanism, as described in the embodiments of this application.

[0108] As one possible implementation, the target 3D point cloud features are obtained. The initial object information and the target 3D point cloud features are used as inputs to the target object information determination model to obtain the target object information, which includes the global target 3D point cloud features of the initial object. Through a trained multi-sensor information fusion model, multiple 2D image features corresponding to each global target 3D point cloud feature of the initial object are determined to obtain fused information.

[0109] It should be understood that the target 3D point cloud features can be the initial 3D point cloud features obtained through a 3D feature extraction model, or they can be 3D point cloud features fused with 2D image features. The specific method for obtaining 3D point clouds fused with 2D image features will be combined with... Figure 14 Please provide a detailed explanation.

[0110] It should also be understood that the embodiments of this application can be applied to other scenarios that require the use of multi-sensor information fusion, such as autonomous driving or robotics, and the embodiments of this application do not limit this.

[0111] In this embodiment, the trained multi-sensor information fusion model is used to fuse 2D image features and 3D point cloud features, so that the relationship between point cloud features and image features can be one-to-many. This means that the fused information is no longer limited to the one-to-one correspondence between 2D image pixels and 3D point clouds determined by the calibration matrix in traditional point-by-point fusion. It can make full use of high-resolution color information and improve the robustness of multi-sensor information fusion.

[0112] The following will combine Figures 6 to 15 right Figure 5 The steps are explained in detail below. For ease of understanding, the explanation is as follows. Figures 6 to 15 The two types of rectangles in the text have different meanings. A "right-angled rectangle" represents the input and output data, while a "rounded rectangle" represents the processing model of the input data, such as processing the input data through a neural network.

[0113] It should be understood that the training and test datasets used in the embodiments of this application are from the large-scale autonomous driving dataset nuScenes. The processing models for the input data described below are models that have been trained using this training dataset, such as 2D feature extraction models, 3D feature extraction models, initial object classification models, target object information determination models, initial object information fusion models, and multi-sensor information fusion models.

[0114] Figure 6 This is a schematic diagram illustrating feature extraction of 3D point cloud information and 2D image information provided in an embodiment of this application. The following will combine... Figure 6 (a) provides a detailed description of acquiring the initial 3D point cloud features, combined with Figure 6 (b) provides a detailed description of S501.

[0115] like Figure 6 As shown in (a), the 3D point cloud information collected by the lidar 203 is used as the input to the 3D feature extraction model to obtain the initial 3D point cloud features. Figure 6The 3D feature extraction model in (a) can be a 3D convolutional neural network for 3D data, or other neural networks for extracting 3D point cloud features, such as PointPillar, a point cloud columnar network for bird eyeview (BEV) columnar structures, sparsely embedded convolutional detection (SECOND) based on sparse networks, or PointNet++, a point cloud network based on original unordered point clouds, etc. The embodiments of this application do not limit this.

[0116] For example, the initial 3D point cloud features can be point cloud bird's-eye view features or other 3D point cloud features. When the 3D object detection box is used for subsequent path planning, since elevation information is not required, point cloud BEV features that ignore elevation information can be used. When the 3D object detection box is used in other scenarios that require elevation information, the 3D mesh can be used to store the point cloud features to retain the height dimension in this embodiment, but this embodiment does not impose any limitations on this. For ease of understanding, this embodiment will subsequently use point cloud BEV features as an example for explanation.

[0117] like Figure 6 As shown in (b), the 2D image information acquired by the camera device 202 is used as the input to the 2D feature extraction model to obtain 2D image features. Figure 6 The 2D feature extraction model in (b) can be a 2D convolutional neural network for 2D data, or other neural networks for extracting 2D image features. This application embodiment does not limit this.

[0118] Figure 7 This is a schematic diagram of an initial object information determination method provided in an embodiment of this application. The following will be combined with... Figure 7 A detailed description of one implementation of S502 is provided.

[0119] like Figure 7 As shown in (a), the target 3D point cloud features are used as input for query initialization to obtain N initial object information, where the initial object information includes the initial object's position information and the initial object's 3D point cloud features, N is a positive integer, and the initial object's position information is the center position information of the initial object. The target 3D point cloud features can be obtained through... Figure 6 (a) The initial 3D point cloud features obtained. The initial object information may also include the category of the initial object. For example, the BEV features of the point cloud are used as input for query initialization to obtain the initial object information. The initial object information may also be called object queries, and the name of the initial object information is not limited in this embodiment.

[0120] For example, such as Figure 7 As shown in (b), the query initialization process can specifically involve using M point cloud BEV features as input to the initial object classifier to obtain M candidate initial object information. The candidate initial object information includes the initial object's location information, the initial object's class probability value, and the initial object's point cloud BEV features, where M is a positive integer. Since the M candidate objects are derived from the BEV two-dimensional feature grid of the point cloud, their class probabilities can be represented as a heatmap. The candidate initial object information can be represented as [x, y, o], where x and y represent the initial object's location information in the bird's-eye view, and o represents the initial object's class information. From the M candidate initial object information, N candidate initial object information with the largest class probability values ​​are selected as N initial object information, where N is a positive integer less than M. For example, the Top-k algorithm can be used to select N initial object information from the M initial object class information. This application embodiment does not limit the specific method of initial object information selection.

[0121] It should be understood that the initial object classifier can be a 2D convolutional layer or other neural networks, and this application embodiment does not limit this. Furthermore, the trained initial object classifier includes predefined object categories; therefore, 3D point cloud features can be used as input to the trained initial object classifier to obtain category-related initial object information.

[0122] It should also be understood that, in the embodiments of this application, Figure 7 The query initialization method shown in (b) is used as an example. Other query initialization methods are also possible, such as the Markov chain Monte Carlo algorithm. This application does not limit the specific methods used.

[0123] In this embodiment, by querying the initialization process, the initial object information used for subsequent 3D object detection is made closer to the center position of the object. Compared with the traditional input-independent initialization method, it is not necessary to go through multiple layers of networks to continuously update the position of the initial object information. Therefore, obtaining fused information based on input-related initial object information can greatly improve the convergence speed and running efficiency of subsequent 3D object detection, improve algorithm efficiency, and reduce network parameters.

[0124] Figure 8 This is a schematic diagram of a multi-sensor information fusion method provided in an embodiment of this application. The following will be combined with... Figure 8 S503 is described in detail.

[0125] like Figure 8As shown, the 2D image features acquired in S501 and the initial object information determined in S502 are used as inputs to the multi-sensor information fusion model to obtain fused information. The multi-sensor information fusion model can be a transformer model, where the initial object information after the self-attention mechanism is used as the parameter query (Q), which is input into the cross-attention mechanism of the multi-sensor information fusion model. The 2D image features are used as the parameter key (K) and the parameter value (V), which are also input into the cross-attention mechanism of the multi-sensor information fusion model.

[0126] It should be understood that the parameter query Q represents the query vector, the parameter key K represents the vector of the queried information in the space of the query vector, and the parameter value V represents the queried information. For example, the initial object information after the self-attention mechanism is used as the query vector, and the 2D image features are used as V and K, which are the vectors of the queried information and the 2D image features in the space of the initial object information after the self-attention mechanism.

[0127] The following will combine Figure 9 The structure of the multi-sensor information fusion model is explained in detail. Figure 9 This is a schematic diagram of the structure of a multi-sensor information fusion model provided in an embodiment of this application. It should be understood that... Figure 9 The diagram illustrates a Transformer model decoding layer structure. Other Transformer model decoding layers are also possible, and this application does not limit this; only one is shown here. Figure 9 Let's take the structure of [example] as an example for explanation.

[0128] like Figure 9 As shown, the first transformer model decoding layer includes P decoding layers, where P is a positive integer. Each decoding layer includes a first multi-head attention and a second multi-head attention. The first multi-head attention is a self-attention mechanism, mainly used to obtain the internal correlation of the initial object information; the second multi-head attention is a cross-attention mechanism, mainly used to obtain the fusion information of the initial object information including 3D point cloud features and 2D image features.

[0129] Specifically, in the first step, the initial object information, along with the initial object's position encoding vector, is input into the first multi-head attention to obtain the output of the first multi-head attention, where the initial object's position encoding vector is the initial object's position information in S502. In the second step, the output of the first multi-head attention is normalized through a skip connection to obtain the first output. In the third step, the first output and the initial object's position encoding vector are added together and input as Q into the second multi-head attention, and the 2D image features are also input into the second multi-head attention as K and V to obtain the output of the second multi-head attention. In the fourth step, the output of the second multi-head attention is normalized through a skip connection to obtain the second output. In the fifth step, the second output is processed through a feedforward process to obtain the third output. In the sixth step, the third output is normalized through a skip connection to obtain fused information, where the fused information integrates the 2D image features and the initial object information, including 3D point cloud features.

[0130] In this embodiment, the cross-attention mechanism in the multi-sensor information fusion model can effectively alleviate the problem of insufficient utilization of high-resolution image information caused by the "hard correlation" relationship between the midpoint cloud and the pixel point in point-by-point fusion, as well as the problem of low robustness of multi-sensor information fusion caused by the use of damaged image features. This improves the robustness of multi-sensor information fusion and the utilization rate of high-resolution images, thereby improving the accuracy of subsequent 3D target detection and recognition as well as the accuracy of target tracking.

[0131] Further, optionally, such as Figure 8 As shown, the fused information is used as input to the target detection box model to obtain the target detection box. The target detection box model can be a feedforward network (FFN), and this embodiment does not limit the type of target detection box model.

[0132] In the above process, the multi-sensor information fusion model directly uses the initial object information and 2D image features as input, which can alleviate the above problems to a certain extent. In order to further improve the richness of the fused information and thus improve the performance of the 3D target detector, the embodiments of this application can further process the initial object information and use the processed initial object information as the input of the multi-sensor information fusion model. The following will combine... Figures 10 to 12 The process of obtaining fused information through a multi-sensor information fusion model in the embodiments of this application will be further described in detail.

[0133] Figure 10 This is a schematic diagram of another multi-sensor information fusion method provided in the embodiments of this application.

[0134] like Figure 10As shown, the 2D image features and target object information obtained in S501 are used as input to the multi-sensor information fusion model. The 2D image features and target object information, including global 3D point cloud features, are fused to obtain fused information, where the target object information includes global 3D point cloud features. Further, and Figure 8 Similar to the example shown, the fused information is used as input to the target detection model to obtain the target detection box, where the target detection model can be a feedforward network.

[0135] For example, the structure and... Figure 9 The models are largely the same, except that the input data for the two multi-sensor information fusion models are different. Figure 10 The first multi-head attention input of the multi-sensor information fusion model in the middle is target object information including global 3D point cloud features, and the second multi-head attention input is... Figure 9 The same as shown.

[0136] In this embodiment, the cross-attention mechanism of the second multi-head attention allows for a flexible point-to-point association between target object information, including global 3D point cloud features, and 2D image features. This ensures that the fusion of image features and point cloud features is unaffected by the number of laser points, fully utilizing high-resolution image information. For example, when the number of laser point clouds is sparse, the 3D point cloud features and 2D image features can have a one-to-many association, thereby fully utilizing high-resolution image information. Furthermore, the target object information including global 3D point cloud features enriches the fused information obtained through the multi-sensor information fusion model, leading to more accurate determination of the subsequent target detection bounding box.

[0137] The following will combine Figure 11 Detailed explanation of the method for obtaining target object information Figure 11 This is a schematic diagram of a method for determining target object information provided in an embodiment of this application.

[0138] like Figure 11As shown, the target 3D point cloud features and the initial object information obtained by S502 are used as inputs to the target object information determination model to obtain the target object information. The target object information determination model can be a transformer model decoding layer. Specifically, the initial object information after the self-attention mechanism is used as Q, which is input into the cross-attention mechanism of the target object information determination model. The 2D image features are used as K and V, which are also input into the cross-attention mechanism of the target object information determination model. For example, when the 3D point cloud features are point cloud BEV features, the initial object information is used as Q, which is the query vector, and the point cloud BEV features are used as V and K, which are the vectors of the query information and the point cloud BEV features in the space where the initial object information after the self-attention mechanism resides.

[0139] The following will combine Figure 12 The structure of the target object information determination model is described in detail. Figure 12 This is a schematic diagram of the structure of a target object information determination model provided in an embodiment of this application. It should be understood that... Figure 12 The diagram illustrates a Transformer model decoding layer structure. Other Transformer model decoding layers are also possible, and this application does not limit this; only one is shown here. Figure 12 Let's take the structure of [example] as an example for explanation.

[0140] like Figure 12 As shown, the target object information determination model includes Q decoding layers, where Q is a positive integer. Each decoding layer includes a first multi-head attention and a second multi-head attention. The first multi-head attention is a self-attention mechanism, mainly used to obtain the internal correlation of the initial object information; the second multi-head attention is a cross-attention mechanism, mainly used to obtain target object information including global 3D point cloud features.

[0141] For detailed steps, please refer to... Figure 9 The description, Figure 12 and Figure 9 The difference lies in the different input of the second multi-head attention.

[0142] As one possible implementation, the target 3D point cloud features, which are input as K and V to the second multi-head attention in the third step, can be obtained through... Figure 6 The initial 3D point cloud features obtained in the manner shown in (a) can also be obtained by... Figure 6 The initial 3D point cloud features obtained in (a) are processed through the encoding layer of the transformer model to obtain enhanced 3D point cloud features. For example, taking the BEV point cloud features as an example, the enhanced BEV point cloud features have a wider feature range and richer information, providing more information for the subsequently obtained object detection boxes.

[0143] The above method can effectively achieve adaptive fusion of point cloud features and image features. However, point cloud features and image features come from different domains. If image features are directly used as input to the multi-sensor information fusion model, a lot of computation time is required to locate the correct image feature region during the fusion process. Therefore, in the embodiments of this application, further improvements have been made to the above scheme to reduce the influence of irrelevant image features during the fusion process, thereby accelerating the rapid convergence of the multi-sensor information fusion model.

[0144] As one possible implementation, the 2D image features obtained by S501 are used as K and V inputs to the second multi-head attention of the point cloud BEV features. This is a spatially modulated cross attention (SMCA) mechanism, which fuses the spatially modulated 2D image features and the point cloud features.

[0145] The spatially confined cross-attention mechanism works by projecting the initial detection box and calibration matrix onto 2D image features, thus limiting the cross-attention range of image features and point cloud features to the initial detection box and filtering out interference from irrelevant background features.

[0146] Specifically, firstly, such as Figure 11 The optional block diagram shown uses the target object information as the output of the model, i.e., the target object information as the input to the initial detection box model, to obtain the initial target detection box. This initial detection box model can be a feedforward network. Subsequently, the initial target detection box and 2D image features are used as input to SMCA, thus limiting the scope of cross-attention to the initial detection box.

[0147] One possible implementation involves using a feedforward network composed of multiple individual convolutional layers to map target object information into initial detection box parameters to form an initial detection box. These initial detection box parameters may include the target object's positional information deviation, the 3D dimensions of the initial detection box, and the yaw angle of the initial detection box. When the 3D point cloud features are from multiple frames, the initial detection box parameters may also include the velocity of the initial detection box. This application does not limit the specific form of the initial detection box parameters.

[0148] Figure 13 This is a spatially constrained cross-attention visualization diagram provided in an embodiment of this application. It should be understood that spatially constrained cross-attention allows attention to image features to be focused on foreground pixels, but high-dimensional image features cannot be visualized. Therefore, Figure 13 This is a spatially constrained visualization of cross-attention on the original image. For example... Figure 13As shown, Figure 13 The wireframe in (a) is the projection wireframe of the initial object information with the initial detection box onto the image. It can be seen that the network can dynamically associate foreground pixels with the initial object information, ignore irrelevant background pixels, and is not affected by the number of laser points. Figure 13 (b) is a visualization of the correlation between the initial object information and the 2D image features when the initial object information and the 2D image features are fused through a spatially constrained cross-attention mechanism. The positions of the lighter-colored dots indicate a high correlation between the initial object information and the 2D image features.

[0149] The embodiments of this application can fuse image features and point cloud features by using a spatially defined cross-attention mechanism that considers both spatial relevance and content relevance. This can effectively filter out relevant image features from the image features and filter out irrelevant or damaged image features, thereby improving the robustness of multi-sensor information fusion and thus improving the robustness of the 3D detector.

[0150] Furthermore, in the above method, the initial detection box is obtained solely based on point cloud features. Since point clouds acquired by LiDAR are inherently sparse, LiDAR collects relatively few point clouds for distant or small objects. If initial object information is obtained solely from initial 3D point cloud features during query initialization, distant or small objects are difficult to detect. Even if image feature fusion is subsequently used to determine the target detection box, image feature fusion cannot be performed on undetected objects, thus affecting not only the determination of the initial detection box but also the subsequent process of determining the target detection box through image feature fusion. To further improve this deficiency, this application addresses it through the following method, which will be discussed below. Figure 14 The specific methods for enhancing the 3D point cloud of a target are explained in detail.

[0151] Figure 14 This is a schematic diagram of a method for enhancing a target 3D point cloud provided in an embodiment of this application.

[0152] like Figure 14 As shown in (a), Figure 6 (b) The 2D image features obtained are used as parameters K and V, and Figure 6 The 3D point cloud features obtained in (a) are input as parameter Q into the cross-attention mechanism to obtain fused 3D point cloud features. Subsequently, according to Figure 7 The query is initialized in the manner shown to obtain the fused initial object information, which, compared with the initial object information, also includes image features.

[0153] For example, taking 3D point cloud as BEV feature, the point cloud features on the BEV plane and the image features on the camera plane are directly cross-attentioned to obtain the fused BEV features.

[0154] In this embodiment of the application, the fused 3D point cloud features include not only 3D point cloud features but also image features, thereby avoiding the limitation of the subsequent determination of the target detection box being restricted by the natural sparsity of the laser point cloud.

[0155] To conserve memory during processing, 2D image features are compressed and preprocessed to reduce their data size. The relationship between the position of the BEV plane and a specific column of the image can be easily established using the calibration matrix between the camera and the LiDAR. For example, the position of the BEV plane and the second dimension W in the image feature [H,W,C] can be easily established using the calibration matrix between the camera and the LiDAR. Typically, at most one object will appear in the H dimension of the image feature; that is, objects are usually distributed horizontally in the image, and only one object usually appears vertically due to occlusion. Therefore, compression can be performed along the H dimension of the image. It should be understood that the horizontal and vertical directions here are absolute concepts and do not change with image rotation.

[0156] For example, such as Figure 14 As shown in (b), the 2D image features are pooled along the high dimension, for example, by max pooling, to obtain compressed image features. Subsequently, the compressed image is input as parameters K and V, and the 3D point cloud features are input as parameters Q into the initial object information fusion model to obtain fused 3D point cloud features. The initial object information fusion model can be a transformer model using only cross-attention. Subsequent steps and... Figure 14 Similar to (a), see reference. Figure 14 The description in (a) will not be repeated here. It should be understood that the embodiments of this application only take the max pooling method as an example to compress 2D image features. Other compression methods can also be used, such as downsampling or average pooling of image features. The embodiments of this application do not limit this.

[0157] In this embodiment of the application, the 2D image features are compressed along the vertical direction to compress the 2D image features, which greatly saves video memory overhead without losing important information.

[0158] The following will combine Figure 15 A complete description of the preferred method for obtaining the target detection bounding box is provided. Figure 15 This is a schematic diagram of a target detection box acquisition method provided in an embodiment of this application.

[0159] like Figure 15 As shown, firstly, 3D point cloud information and 2D image information are extracted using 3D feature extraction models and 2D feature extraction models, respectively, to obtain 3D point cloud features and 2D image features. Secondly, in the initial object information determination process, the 3D point cloud features can be directly used as input for query initialization to obtain the initial object information, or the 3D point cloud features fused with 2D image features can be used as input for query initialization to obtain the initial object information. Subsequently, in the target object information determination process, the initial object information and 3D point cloud features can be used as input to the target object information determination model to obtain the target object information; or, the initial object information and the fused 3D point cloud features can be used as input to the target object information determination model to obtain the target object information. Subsequently, in the information fusion process, the target object information and 2D image features can be directly used as input to the multi-sensor information fusion model to obtain the fused information; or, the initial detection box obtained based on the target object information can be used as the spatial constraint condition of the cross-attention mechanism and input into the multi-sensor information fusion model to fuse the target object information and 2D image features to obtain the fused information. Finally, during the object detection box determination process, the fused information is used as input to the object detection box model to obtain the object detection box.

[0160] In the embodiments of this application, the three-dimensional target detector using the multi-sensor information fusion method described above makes the hard correlation mode between image pixels and point clouds no longer one-to-one. Therefore, under abnormal image conditions (e.g., the image shooting lighting conditions are too good or too bad, image features are damaged, and the camera and lidar calibration matrix is ​​inaccurate), the three-dimensional target detector maintains good robustness and makes full use of image features, making three-dimensional target detection more accurate.

[0161] It should be understood that the fused information obtained by the multi-sensor information fusion method in the embodiments of this application can be used not only for 3D target detection, but also for other downstream tasks, such as target tracking and target recognition. The embodiments of this application do not limit this.

[0162] The above content describes the multi-sensor information fusion method according to an embodiment of this application. The following will combine... Figures 16 to 18 A detailed description of the multi-sensor information fusion device is provided. It should be understood that the descriptions of the device embodiments and the method embodiments correspond to each other. Therefore, any content not described in detail can be found in the method embodiments above, and for the sake of brevity, will not be repeated here.

[0163] Figure 16 This is a schematic diagram of a multi-sensor information fusion device provided in an embodiment of this application. The device includes: an acquisition unit 1601, a determination unit 1602, and a fusion unit 1603.

[0164] The acquisition unit 1601 is used to acquire 2D image features;

[0165] The determining unit 1602 is used to determine initial object information, wherein the initial object information includes the position information of the initial object and the 3D point cloud features of the initial object;

[0166] The fusion unit 1603 is used to determine multiple 2D image features corresponding to each 3D point cloud feature of the initial object through a trained multi-sensor information fusion model, so as to obtain fused information.

[0167] It should be understood that the above description is merely illustrative. This multi-sensor information fusion device is used to execute the methods or steps mentioned in the foregoing method embodiments. Therefore, this device corresponds to the foregoing method embodiments. For details, please refer to the description of the foregoing method embodiments, which will not be repeated here.

[0168] Figure 17 This is a schematic diagram of the hardware structure of a multi-sensor information fusion device provided in this application. Figure 17 The multi-sensor information fusion device 1700 shown may include a memory 1710, a processor 1720, and a communication interface 1730. The memory 1710, processor 1720, and communication interface 1730 are connected via an internal connection path. The memory 1710 stores instructions, and the processor 1720 executes the instructions stored in the memory 1720 to control the input / output interface 1730 to receive / transmit at least some parameters of the second channel model. Optionally, the memory 1710 may be coupled to the processor 1720 via an interface, or it may be integrated with the processor 1720.

[0169] It should be noted that the aforementioned communication interface 1730 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the communication device 1700 and other devices or communication networks. The aforementioned communication interface 1730 may also include an input / output interface.

[0170] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1720 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1710, and the processor 1720 reads the information in memory 1710 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0171] It should be understood that in the embodiments of this application, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0172] It should also be understood that, in embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0173] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0174] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0175] Figure 18 A schematic diagram of a system 1800 provided in an embodiment of this application is shown. For example... Figure 18 As shown, the system includes multiple sensors 1801 and a multi-sensor information fusion device 1802, wherein the multi-sensor information fusion device 1802 may be the aforementioned multi-sensor information fusion device 1700.

[0176] Optionally, the multi-sensor information fusion device may be located in a cloud server.

[0177] Optionally, if the multi-sensor information fusion device is a multi-sensor information fusion device located in a cloud server, the system 1800 further includes a means for receiving instructions from the multi-sensor information fusion device 1802.

[0178] This application also provides an apparatus comprising a processing unit and a storage unit, wherein the storage unit stores instructions, and the processing unit executes the instructions stored in the storage unit to enable the apparatus to perform the aforementioned multi-sensor information fusion method.

[0179] This application also provides a computer-readable medium storing program code that, when executed on a computer, causes the computer to perform the above-described actions. Figures 3 to 10 Any of the methods mentioned above.

[0180] This application also provides a chip, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figures 3 to 10 Any of the methods mentioned above.

[0181] This application also provides a vehicle, including: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory to perform the above-mentioned... Figures 3 to 10 Any of the methods mentioned above.

[0182] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0183] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different devices to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the apparatus described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-sensor information fusion method, characterized by, The method comprises the following steps: obtaining 2D image features; determining initial object information, wherein the initial object information comprises position information of an initial object and 3D point cloud features of the initial object; determining, by using a trained multi-sensor information fusion model, a plurality of 2D image features corresponding to each of the 3D point cloud features of the initial object to obtain fusion information. The method comprises the following steps: obtaining target 3D point cloud features; performing a query initialization operation on the target 3D point cloud features to determine initial object information, wherein the position information of the initial object is the center position information of the initial object, and the 3D point cloud features of the initial object are the target 3D point cloud features of the initial object.

2. The method of claim 1, wherein, The trained multi-sensor information fusion model comprises a cross-attention mechanism.

3. The method according to claim 1 or 2, characterized in that, The method comprises the following steps: taking the target 3D point cloud features as input of an initial object classification model to obtain M candidate initial object information, wherein the candidate initial object information comprises class probability values of the candidate initial objects; determining N of the M candidate initial object information as the initial object information according to the class probability values of the M candidate initial objects, wherein N is a positive integer less than M, and M is a positive integer.

4. The method according to claim 1 or 2, characterized in that, The method comprises the following steps: taking the initial object information and the target 3D point cloud features as input of a target object information determination model to obtain target object information, wherein the target object information comprises global target 3D point cloud features of the initial object; determining, by using the trained multi-sensor information fusion model, a plurality of 2D image features corresponding to each of the global target 3D point cloud features of the initial object to obtain the fusion information.

5. The method of claim 4, wherein, The method comprises the following steps: taking the target object information as input of an initial detection frame model to obtain an initial detection frame; determining, by using the trained multi-sensor information fusion model, a plurality of 2D image features corresponding to each of the global target 3D point cloud features of the initial object to obtain the fusion information, wherein the spatially defined cross-attention mechanism in the trained multi-sensor information fusion model is a cross-attention mechanism defined by the initial detection frame, and the spatially defined cross-attention mechanism is used for multi-sensor information fusion after spatial limitation of the 2D image features.

6. The method of claim 1 or 2, wherein, The method comprises the following steps: obtaining initial 3D point cloud features; taking the 2D image features and the initial 3D point cloud features as input of an initial object information fusion model to obtain the target 3D point cloud features, wherein the target 3D point cloud features are 3D point cloud features fused with the 2D image features.

7. The method of claim 6, wherein, The 2D image feature is a 2D image feature after longitudinal compression.

8. The method of claim 1 or 2, wherein, The target 3D point cloud feature is a target bird's eye view (BEV) feature.

9. A multi-sensor information fusion device, characterized by comprising: The device comprises an acquisition unit, a determination unit and a fusion unit, which comprises: The acquisition unit is configured to acquire a 2D image feature. The determination unit is configured to determine initial object information, wherein the initial object information comprises position information of an initial object and a 3D point cloud feature of the initial object. The fusion unit is configured to determine a plurality of 2D image features corresponding to each 3D point cloud feature of the initial object by using the trained multi-sensor information fusion model, so as to obtain fusion information. The acquisition unit is configured to acquire a target 3D point cloud feature. The determination unit is configured to perform a query initialization operation on the target 3D point cloud feature, and determine initial object information, wherein the position information of the initial object is the center position information of the initial object, and the 3D point cloud feature of the initial object is the target 3D point cloud feature of the initial object.

10. The apparatus of claim 9, wherein, The trained multi-sensor information fusion model comprises a cross-attention mechanism.

11. The apparatus of claim 9 or 10, wherein, The determination unit is specifically configured to: use the target 3D point cloud feature as an input of an initial object classification model, to obtain M candidate initial object information, wherein the candidate initial object information comprises a class probability value of a candidate initial object; determine N of the M candidate initial object information as the initial object information according to the class probability values of the M candidate initial objects, wherein N is a positive integer less than M, and M is a positive integer.

12. The apparatus of claim 9 or 10, wherein, The fusion unit is specifically configured to: use the initial object information and the target 3D point cloud feature as an input of a target object information determination model, to obtain target object information, wherein the target object information comprises a global target 3D point cloud feature of the initial object; determine a plurality of 2D image features corresponding to each global target 3D point cloud feature of the initial object by using the trained multi-sensor information fusion model, so as to obtain the fusion information.

13. The apparatus of claim 12, wherein, The fusion unit is specifically configured to: use the target object information as an input of an initial detection frame model, to obtain an initial detection frame; determine a plurality of 2D image features corresponding to each global target 3D point cloud feature of the initial object by using the trained multi-sensor information fusion model, so as to obtain the fusion information, wherein a spatially defined cross-attention mechanism in the trained multi-sensor information fusion model is a cross-attention mechanism defined by the initial detection frame, and the spatially defined cross-attention mechanism is used for spatially defined multi-sensor information fusion of the 2D image feature.

14. The apparatus of claim 9 or 10, wherein, The acquisition unit is specifically configured to: acquire an initial 3D point cloud feature; use the 2D image feature and the initial 3D point cloud feature as an input of an initial object information fusion model, to obtain the target 3D point cloud feature, wherein the target 3D point cloud feature is a 3D point cloud feature fused with the 2D image feature.

15. The apparatus of claim 14, wherein, The 2D image feature is a 2D image feature after longitudinal compression.

16. The apparatus of claim 9 or 10, wherein, The target 3D point cloud feature is a target bird's eye view (BEV) feature.

17. A multi-sensor information fusion device, characterized by comprising: The device comprises a processor and a memory, the memory is used to store program instructions, and the processor is used to call the program instructions to execute the method in any one of claims 1 to 8.

18. A vehicle characterized by comprising: The device comprises any one of claims 9 to 16.

19. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, and when the program instructions are run by a processor, the method in any one of claims 1 to 8 is implemented.

20. A system, comprising: The device comprises a plurality of sensors and a multi-sensor information fusion device, wherein the multi-sensor information fusion device is any one of claims 9 to 16.

Images