Multi-sensor fusion sensing method and device, terminal and storage medium
By using a multi-sensor fusion perception model, which leverages complementary information from surround view images, radar information, and lidar point clouds, high-precision target perception and position estimation are achieved. This solves the problem of low accuracy in complex environments caused by traditional multi-sensor fusion algorithms, thereby improving the safety and robustness of autonomous driving.
Patent Information
- Application Number
- CN202411050581.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-26
AI Technical Summary
Traditional multi-sensor fusion algorithms have low accuracy in target perception and position estimation under complex road and environmental conditions, and single sensors perform poorly in adverse weather conditions, failing to guarantee the safety and accuracy of autonomous driving.
By acquiring panoramic images, radar information, and lidar point clouds, a multi-sensor fusion perception model is used to perform information complementarity and fusion, including a visual perception module, an MLP network, a PillarNet backbone network, and a multi-sensor fusion module. A transformer network is used for feature fusion to output the category, location, and velocity information of the perceived target.
Achieving high-precision target perception and position estimation under various complex road and environmental conditions improves the safety and robustness of autonomous driving and solves the problem of insufficient perception by a single sensor in adverse weather conditions.
Smart Images

Figure CN119026074A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multi-sensor fusion perception method, device, terminal and storage medium. Background Art
[0002] With the development of autonomous driving technology, autonomous driving has gradually become an indispensable basic software function of automobiles. At the same time, the problems faced by autonomous driving are becoming increasingly prominent.
[0003] There are many sensors used in traditional autonomous driving, especially cameras, radars, and lidars. However, there are many limitations to relying on a single sensor. For example, a certain model that uses a camera sensor alone cannot enter autonomous driving in rainy, foggy weather and poor lighting conditions. In addition, the prediction accuracy of the distance and speed of objects of this model is not high, and the safety of autonomous driving cannot be guaranteed. The lidar sensors used in most L4 intelligent driving solutions also have some problems with rain and fog noise and reflection misdetection, as well as insufficient perception distance due to the obstruction or absorption of the lidar point cloud in heavy rain and fog. In addition, models that only use radar sensors can only capture the position and speed information of objects, lack texture and semantic information, resulting in limited perception functions. Therefore, fusing data from multiple sensors can improve the robustness and accuracy of the perception system. However, traditional multi-sensor fusion algorithms have problems with data redundancy and high computational complexity, resulting in low accuracy of target perception and position estimation under various complex road and environmental conditions.
[0004] Therefore, the prior art needs to be improved. Summary of the invention
[0005] The technical problem to be solved by the present invention is that, in view of the defects of the prior art, the present invention provides a multi-sensor fusion perception method, device, terminal and storage medium to solve the problem of low accuracy of target perception and position estimation of traditional multi-sensor fusion algorithms under various complex road and environmental conditions.
[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0007] In a first aspect, the present invention provides a multi-sensor fusion perception method, comprising:
[0008] Obtain surround view images, radar information, and lidar point clouds;
[0009] Based on a multi-sensor fusion perception model, the surround view image, the radar information and the complementary information of the lidar point cloud are used for fusion and perception, and the category, position, size and speed information of the perceived target are output.
[0010] In one implementation, the acquiring of surround view images, radar information, and lidar point cloud includes:
[0011] Get the current frame surround image collected by the surround camera, the current frame radar information collected by the radar sensor, and the current frame lidar point cloud collected by the lidar.
[0012] In one implementation, the multi-sensor fusion perception model includes: a visual perception module, an MLP network, a pillarnet backbone network, and a multi-sensor fusion module;
[0013] Among them, the visual perception module includes: a CNN network and an FPN network; the multi-sensor fusion module includes: a first transformer network, a second transformer network and a third transformer network.
[0014] In one implementation, the multi-sensor fusion perception model is based on the use of complementary information of the surround view image, the radar information, and the lidar point cloud for fusion and perception, including:
[0015] Inputting the surround view image of the current frame into the visual perception module, and processing it through the CNN network and the FPN network to extract image features of different scales of each image;
[0016] Inputting the current frame radar information into the MLP network to extract radar information;
[0017] Input the current frame laser radar point cloud into the pillarnet backbone network to obtain the laser radar BEV feature;
[0018] Based on the multi-sensor fusion module, the image features, the radar information and the lidar BEV features are fused to output the category, position, size and speed information of the perceived target.
[0019] In one implementation, the fusing of the image features, the radar information, and the lidar BEV features based on the multi-sensor fusion module includes:
[0020] The detection query is randomly initialized in a predefined detection space to obtain an initialized detection query.
[0021] In one implementation, the fusing of the image features, the radar information, and the lidar BEV features based on the multi-sensor fusion module includes:
[0022] Inputting the initialized detection query and the extracted radar information into the first transformer network for fusion, updating the position information and speed information in the initialized detection query, and obtaining an updated first detection query;
[0023] Inputting the first detection query and the extracted image features into the second transformer network for fusion, projecting the coordinate information in the first detection query to each camera, extracting the image feature information in each camera, and obtaining an updated second detection query;
[0024] Input the second detection query and the extracted laser radar BEV feature into the third transformer network for fusion, and update the position information and feature information of the second detection query according to the laser radar BEV feature to obtain an updated third detection query;
[0025] Wherein, the third detection query includes: category, position, size and speed information of the perceived target.
[0026] In one implementation, the updating of the location information and feature information of the second detection query according to the laser radar BEV feature includes:
[0027] Based on the coordinate information and length, width, and height information of the second detection query, sampling is performed in the laser radar BEV features, and the position information and feature information of the second detection query are updated according to the sampled features.
[0028] In a second aspect, the present invention provides a multi-sensor fusion perception device, comprising:
[0029] Acquisition module, used to acquire surround view images, radar information and lidar point cloud;
[0030] The fusion and perception module is used to fuse and perceive the surround view image, the radar information and the complementary information of the lidar point cloud based on a multi-sensor fusion perception model, and output the category, position, size and speed information of the perceived target.
[0031] In a third aspect, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a multi-sensor fusion perception program, and when the multi-sensor fusion perception program is executed by the processor, it is used to implement the operation of the multi-sensor fusion perception method as described in the first aspect.
[0032] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, and which stores a multi-sensor fusion perception program. When the multi-sensor fusion perception program is executed by a processor, it is used to implement the operation of the multi-sensor fusion perception method as described in the first aspect.
[0033] The present invention adopts the above technical solution to achieve the following effects:
[0034] By acquiring surround view images, radar information and lidar point clouds, the present invention can fuse and perceive them based on a multi-sensor fusion perception model, using the complementary information of surround view images, radar information and lidar point clouds, and output the category, position, size and speed information of the perceived target; by making full use of the complementary information of radar, camera and lidar, the present invention can achieve high-precision target perception and position estimation under various complex road and environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.
[0036] Figure 1 It is a flow chart of a multi-sensor fusion perception method in one implementation of the present invention.
[0037] Figure 2 It is a structural diagram of a multi-sensor fusion perception model in one implementation of the present invention.
[0038] Figure 3 It is a processing schematic diagram of a multi-sensor fusion perception model in one implementation of the present invention.
[0039] Figure 4 It is a functional principle diagram of a terminal in one implementation of the present invention.
[0040] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0042] Exemplary Methods
[0043] There are many sensors used in traditional autonomous driving, especially cameras, radars, and lidars. However, there are many limitations to relying on a single sensor. For example, a certain model that uses a camera sensor alone cannot enter autonomous driving in rainy, foggy weather and poor lighting conditions. In addition, the prediction accuracy of the distance and speed of objects of this model is not high, and the safety of autonomous driving cannot be guaranteed. The lidar sensors used in most L4 intelligent driving solutions also have some problems with rain and fog noise and reflection misdetection, as well as insufficient perception distance due to the obstruction or absorption of the lidar point cloud in heavy rain and fog. In addition, models that only use radar sensors can only capture the position and speed information of objects, lack texture and semantic information, resulting in limited perception functions. Therefore, fusing data from multiple sensors can improve the robustness and accuracy of the perception system. However, traditional multi-sensor fusion algorithms have problems with data redundancy and high computational complexity, resulting in low accuracy of target perception and position estimation under various complex road and environmental conditions.
[0044] In response to the above technical problems, an embodiment of the present invention provides a multi-sensor fusion perception method, which acquires surround view images, radar information and lidar point clouds, and can be based on a multi-sensor fusion perception model, and utilizes the complementary information of surround view images, radar information and lidar point clouds for fusion and perception, and outputs the category, position, size and speed information of the perceived target; the present invention makes full use of the complementary information of radar, camera and lidar, and can achieve high-precision target perception and position estimation under various complex road and environmental conditions.
[0045] like Figure 1 As shown, an embodiment of the present invention provides a multi-sensor fusion perception method, comprising the following steps:
[0046] Step S100, obtaining surround view images, radar information and lidar point cloud.
[0047] In this embodiment, in order to realize the multi-sensor fusion perception method, a fusion algorithm based on sparsefusion (i.e., multimodal fusion) is proposed, which is a radar, camera, and lidar multi-sensor fusion perception algorithm based on sparse fusion. The algorithm effectively fuses data from three different sensors to achieve the effect of complementing each other's strengths, thereby providing more accurate and robust environmental perception and target detection capabilities. By making full use of the complementary information of radar, camera, and lidar, the algorithm can achieve high-precision target perception and position estimation under various complex road and environmental conditions.
[0048] Specifically, in an implementation of this embodiment, step S100 includes the following steps:
[0049] Step S101, obtaining the current frame surround image collected by the surround camera, the current frame radar information collected by the radar sensor, and the current frame lidar point cloud collected by the lidar.
[0050] In this embodiment, in an autonomous driving scenario, when the vehicle enters a scene (for example, a road driving scenario, a parking lot driving and parking scenario), it is necessary to perceive all targets in the current vehicle environment (for example, obstacles, pedestrians, buildings, and other moving objects, etc.), and estimate the positions and speeds corresponding to these targets, so as to perform autonomous driving planning based on the perception results of these targets and determine a drivable path.
[0051] In this embodiment, when the vehicle enters a driving scene, the surround view image of the current scene can be obtained through the on-board surround view camera, the radar information of the target in the current scene (for example, the position and speed of the target) can be obtained through the on-board radar sensor, and the lidar point cloud data of the target in the current scene can be obtained through the on-board lidar sensor.
[0052] In order to synchronize the information obtained by the three sensors, when the vehicle enters a driving scene, this embodiment aligns the data of the three sensors by acquiring the current frame surround image collected by the surround view camera, the current frame radar information collected by the radar sensor, and the current frame lidar point cloud collected by the lidar at the same moment.
[0053] As an example, the surround view images in this embodiment are collected by surround view fisheye cameras installed under the left and right rearview mirrors and under the front and rear license plates of the vehicle. The surround view fisheye camera has a large viewing angle of more than 180° and has better perception at close distances; the lidar point cloud is collected by a lidar installed on the top of the vehicle; and the radar information is collected by radar sensors installed on the front and rear bumpers of the vehicle.
[0054] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multi-sensor fusion perception method further includes the following steps:
[0055] Step S200, based on a multi-sensor fusion perception model, the surround view image, the radar information and the complementary information of the lidar point cloud are used for fusion and perception, and the category, position, size and speed information of the perceived target are output.
[0056] In this embodiment, for the acquired current frame surround view image, current frame radar information and current frame lidar point cloud, a multi-sensor fusion perception model can be used to extract corresponding information and complement information therefrom, so as to solve the problem of insufficient performance of a single sensor in different scenarios; for example, in severe rainy and snowy weather, cameras and radars can be used to improve the problem of insufficient detection distance of lidar; through lidar or radar, the position accuracy and speed accuracy of camera detection of targets can be improved.
[0057] like Figure 2 As shown, in one implementation of an embodiment of the present invention, the multi-sensor fusion perception model includes: a visual perception module, an MLP (Multilayer Perceptron) network, a pillarnet (pillar-based single-stage 3D object detector) backbone network and a multi-sensor fusion module.
[0058] Among them, the visual perception module includes: a CNN network (Convolutional Neural Networks) and an FPN network (Feature Pyramid Networks); the CNN network and the FPN network are used to extract features of different scales of each image in the surround image, so as to realize the fusion and utilization of multi-scale features and improve the target detection accuracy; the MLP network is a feedforward neural network, which is used to perceive the radar information and obtain the radar information of the target; the pillarnet backbone network is used to perceive the target of the lidar point cloud and obtain the lidar BEV features to realize 3D target detection; the multi-sensor fusion module is used to fuse the features and information extracted by the visual perception module, the MLP network and the pillarnet backbone network, so as to make full use of the complementary information of radar, camera and lidar.
[0059] Among them, the multi-sensor fusion module includes: a first transformer network, a second transformer network and a third transformer network; each of the transformer networks includes a self-attention layer and a cross-attention layer.
[0060] Specifically, in an implementation of this embodiment, step S200 includes the following steps:
[0061] Step S201, inputting the surround view image of the current frame into the visual perception module, and processing it through the CNN network and the FPN network to extract image features of different scales of each image;
[0062] Step S202, inputting the current frame radar information into the MLP network to extract radar information;
[0063] Step S203, inputting the current frame laser radar point cloud into the pillarnet backbone network to obtain the laser radar BEV feature;
[0064] Step S204: Based on the multi-sensor fusion module, the image features, the radar information and the lidar BEV features are fused to output the category, position, size and speed information of the perceived target.
[0065] like Figure 3 As shown, in this embodiment, the acquired surround view image of the current frame is input into the CNN backbone network + FPN module (i.e., the visual perception module), and the image features of different scales of each image are output through the CNN network and the FPN network therein to obtain the camera-based target detection result of the current scene.
[0066] For the current frame radar information (for example, the position and speed information of the target), it is input into the MLP network, and after multi-layer perception, a more accurate radar-based target detection result is obtained. For the current frame lidar point cloud, it is input into the pillarnet backbone network, and the lidar BEV features are extracted to obtain the lidar-based target detection result.
[0067] In this embodiment, the image features, the radar information and the lidar BEV features are fused based on the multi-sensor fusion module, and the extracted features and information are complemented by the self-attention mechanism and the cross-attention mechanism, so as to accurately output the category, position, size and speed information of the perceived target, thereby achieving high-precision target perception and position estimation under various complex road and environmental conditions.
[0068] Specifically, in an implementation of this embodiment, the fusion of the image features, the radar information and the lidar BEV features based on the multi-sensor fusion module includes: randomly initializing the detection query in a predefined detection space to obtain an initialized detection query.
[0069] like Figure 3 As shown, in this embodiment, before fusing the image features, the radar information and the lidar BEV features, detection query initialization processing is required, specifically: randomly initializing a detection query in a predefined detection space, and the initialized detection query contains 256-dimensional position encoding information and 256-dimensional feature information.
[0070] Specifically, in an implementation of the present embodiment, the image features, the radar information and the lidar BEV features are fused based on the multi-sensor fusion module, including: inputting the initialized detection query and the extracted radar information into the first transformer network for fusion, updating the position information and speed information in the initialized detection query, and obtaining an updated first detection query; inputting the first detection query and the extracted image features into the second transformer network for fusion, projecting the coordinate information in the first detection query into each camera, extracting image feature information in each of the cameras, and obtaining an updated second detection query; inputting the second detection query and the extracted lidar BEV features into the third transformer network for fusion, updating the position information and feature information of the second detection query according to the lidar BEV features, and obtaining an updated third detection query.
[0071] like Figure 3 As shown, in this embodiment, for the initialized detection query, it is first fused with the radar information output by the MLP network through a transformer module to update the position and speed information in the initialized detection query.
[0072] The specific fusion process is as follows: first, the initialized detection query is weighted using the self-attention mechanism; then, the weighted detection query is fused with the radar information output by the MLP network, and the fusion process is weighted using the cross-attention mechanism to obtain the updated first detection query.
[0073] For the updated first detection query, it is input into a transformer with the camera features (i.e., the image features extracted by the CNN backbone network + FPN module) for fusion; wherein, the fusion method is to combine the internal and external parameters of the camera, project the xyz coordinate information in the first detection query to each camera, extract the image feature information in the camera, and then update the second detection query.
[0074] The specific fusion process is as follows: first, the updated first detection query is weighted using the self-attention mechanism; then, based on the multi-scale image features output by the CNN backbone network + FPN module, the camera internal and external parameters are introduced to query the image features; finally, the weighted detection query is fused with the image features of the camera internal and external parameter query, and the fusion process is weighted using the cross-attention mechanism to obtain the updated second detection query.
[0075] For the updated second detection query, it is fused with the lidar BEV feature and also input into the transformer structure. The position information (xyz information) and length, width and height information (lwh information) of the updated second detection query are directly used to sample features in the lidar BEV feature, and the position information and feature information of the second detection query are updated to obtain an updated third detection query; wherein, the third detection query includes: the category, position, size and speed information of the perceived target.
[0076] The specific fusion process is as follows: first, the updated second detection query is weighted using the self-attention mechanism; then, the lidar BEV features output by the pillarnet backbone network are sampled; finally, the weighted detection query is fused with the sampled features, and the fusion process is weighted using the cross-attention mechanism to obtain the updated third detection query.
[0077] Specifically, in one implementation of the present embodiment, the updating of the position information and feature information of the second detection query according to the laser radar BEV feature includes: sampling in the laser radar BEV feature based on the coordinate information and the length, width, and height information of the second detection query, and updating the position information and feature information of the second detection query according to the sampled features.
[0078] In this embodiment, the fusion process of the multi-sensor fusion module (i.e., the fusion processing of the three transformer modules) can be iterated multiple times until the detection speed meets the requirements; for example, it can be repeated 1 to n times, depending on the required operating speed, and the more times, the better the effect.
[0079] In this embodiment, during the training phase of the multi-sensor fusion perception model, the Hungarian matching algorithm is used to allocate positive and negative samples between the detection query and gt (gt refers to a set of manually labeled real object bounding boxes and category information), and a loss training network is constructed according to the allocated query-gt pairs.
[0080] It is worth mentioning that the fusion algorithm based on sparse fusion (i.e., multimodal fusion) proposed in this embodiment supports the output of complete perception when some sensors are missing, which is of great help to the safety and stability of autonomous driving. This embodiment is based on the sparse fusion architecture to efficiently fuse the features of multiple sensors, so that the network is efficient and fast. The fusion process is carried out in the network. Compared with the post-fusion method, a lot of post-processing and some rule definitions are reduced, allowing the network to learn in the data and achieve a better end2end effect.
[0081] In summary, this embodiment improves the robustness of the perception algorithm and the scene adaptability by fusing radar, camera, and lidar, and solves the drawbacks of a single sensor; and uses multimodal fusion to fuse the features of different sensors, and performs sparse feature extraction on the preprocessed data. By selecting the most representative data points or features, data redundancy is reduced, and key information is retained, so that the detection pays more attention to the obstacle area, avoids ineffective computing power investment in the background area, and achieves more efficient use of computing power and improves speed.
[0082] This embodiment achieves the following technical effects through the above technical solution:
[0083] By acquiring surround view images, radar information, and lidar point clouds, this embodiment can fuse and perceive the surround view images, radar information, and complementary information of lidar point clouds based on a multi-sensor fusion perception model, and output the category, position, size, and speed information of the perceived target; by making full use of the complementary information of radar, camera, and lidar, this embodiment can achieve high-precision target perception and position estimation under various complex road and environmental conditions.
[0084] Exemplary Devices
[0085] Based on the above embodiments, the present invention further provides a multi-sensor fusion sensing device, comprising:
[0086] Acquisition module, used to acquire surround view images, radar information and lidar point cloud;
[0087] The fusion and perception module is used to fuse and perceive the surround view image, the radar information and the complementary information of the lidar point cloud based on a multi-sensor fusion perception model, and output the category, position, size and speed information of the perceived target.
[0088] This embodiment achieves the following technical effects through the above technical solution:
[0089] By acquiring surround view images, radar information, and lidar point clouds, this embodiment can fuse and perceive the surround view images, radar information, and complementary information of lidar point clouds based on a multi-sensor fusion perception model, and output the category, position, size, and speed information of the perceived target; by making full use of the complementary information of radar, camera, and lidar, this embodiment can achieve high-precision target perception and position estimation under various complex road and environmental conditions.
[0090] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Figure 4 shown.
[0091] The terminal includes: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.
[0092] When the computer program is executed by a processor, it is used to implement the operation of the multi-sensor fusion perception method.
[0093] It can be understood by those skilled in the art that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0094] In one embodiment, a terminal is provided, which includes: a processor and a memory, wherein the memory stores a multi-sensor fusion perception program, and the multi-sensor fusion perception program is used to implement the operation of the above multi-sensor fusion perception method when executed by the processor.
[0095] In one embodiment, a storage medium is provided, wherein the storage medium stores a multi-sensor fusion perception program, and the multi-sensor fusion perception program is used to implement the operation of the above multi-sensor fusion perception method when executed by a processor.
[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and volatile memory.
[0097] In summary, the present invention provides a multi-sensor fusion perception method, device, terminal and storage medium, the method comprising: acquiring surround view images, radar information and lidar point cloud; based on a multi-sensor fusion perception model, utilizing the complementary information of the surround view images, the radar information and the lidar point cloud for fusion and perception, and outputting the category, position, size and speed information of the perceived target; the present invention can achieve high-precision target perception and position estimation under various complex road and environmental conditions by making full use of the complementary information of radar, camera and lidar.
[0098] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multi-sensor fusion perception method, characterized in that: include: Obtain surround view images, radar information, and lidar point clouds; Based on the multi-sensor fusion perception model, the surround view image, the radar information and the complementary information of the lidar point cloud are used for fusion and perception, and the category, position, size and speed information of the perceived target are output.
2. The multi-sensor fusion perception method according to claim 1, characterized in that: The obtaining of surround view images, radar information and lidar point cloud includes: Get the current frame surround image collected by the surround camera, the current frame radar information collected by the radar sensor, and the current frame lidar point cloud collected by the lidar.
3. The multi-sensor fusion perception method according to claim 1, characterized in that: The multi-sensor fusion perception model includes: a visual perception module, an MLP network, a pillarnet backbone network and a multi-sensor fusion module; Among them, the visual perception module includes: a CNN network and an FPN network; the multi-sensor fusion module includes: a first transformer network, a second transformer network and a third transformer network.
4. The multi-sensor fusion perception method according to claim 3, characterized in that: The multi-sensor fusion perception model is based on the use of complementary information of the surround view image, the radar information and the laser radar point cloud for fusion and perception, including: Inputting the surround view image of the current frame into the visual perception module, and processing it through the CNN network and the FPN network to extract image features of different scales of each image; Inputting the current frame radar information into the MLP network to extract radar information; Input the current frame laser radar point cloud into the pillarnet backbone network to obtain the laser radar BEV feature; Based on the multi-sensor fusion module, the image features, the radar information and the lidar BEV features are fused to output the category, position, size and speed information of the perceived target.
5. The multi-sensor fusion perception method according to claim 4, characterized in that: The fusing of the image features, the radar information and the laser radar BEV features based on the multi-sensor fusion module comprises: The detection query is randomly initialized in a predefined detection space to obtain an initialized detection query.
6. The multi-sensor fusion perception method according to claim 4, characterized in that: The fusing of the image features, the radar information and the laser radar BEV features based on the multi-sensor fusion module includes: Inputting the initialized detection query and the extracted radar information into the first transformer network for fusion, updating the position information and speed information in the initialized detection query, and obtaining an updated first detection query; Inputting the first detection query and the extracted image features into the second transformer network for fusion, projecting the coordinate information in the first detection query to each camera, extracting the image feature information in each camera, and obtaining an updated second detection query; Input the second detection query and the extracted laser radar BEV feature into the third transformer network for fusion, and update the position information and feature information of the second detection query according to the laser radar BEV feature to obtain an updated third detection query; Wherein, the third detection query includes: category, position, size and speed information of the perceived target.
7. The multi-sensor fusion perception method according to claim 6, characterized in that: The updating of the location information and feature information of the second detection query according to the laser radar BEV feature includes: Based on the coordinate information and length, width, and height information of the second detection query, sampling is performed in the laser radar BEV features, and the position information and feature information of the second detection query are updated according to the sampled features.
8. A multi-sensor fusion sensing device, characterized in that: include: Acquisition module, used to acquire surround view images, radar information and lidar point cloud; The fusion and perception module is used to fuse and perceive the surround view image, the radar information and the complementary information of the lidar point cloud based on a multi-sensor fusion perception model, and output the category, position, size and speed information of the perceived target.
9. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a multi-sensor fusion perception program, and when the multi-sensor fusion perception program is executed by the processor, it is used to implement the operation of the multi-sensor fusion perception method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-sensor fusion perception program, which, when executed by a processor, is used to implement the operation of the multi-sensor fusion perception method according to any one of claims 1 to 7.
Citation Information
Cited By
Endogenous security network elastic evaluation and judgment method, system and device and storage medium
CN119249120A
Traffic target tracking method and system, electronic equipment and storage medium
CN121259048A
