Three-dimensional target detection and recognition method based on multi-source fusion in vehicle-road cooperative scenario

Abstract

The present invention is a three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road collaboration scenario, and relates to the field of vehicle networking and intelligent vehicle technology. In the present invention, different environmental information is captured by sensors of different roadside equipment, and multimodal features for three-dimensional target detection are extracted and transmitted to the roadside feature fusion center, which fuses the obtained multi-channel multimodal features into multi-source fusion features; the intelligent vehicle calibrates and fuses the multi-source fusion features provided by the roadside and the multimodal features extracted by its own perception, and further performs target recognition and detection. In the present invention, the point cloud feature extraction module adopts a neighborhood adaptive feature extraction method, and the feature calibration and fusion module performs feature-level fusion of the point cloud feature map and the image feature map based on the meta-learning method. The present invention greatly improves the perception of the external environment, reduces the communication pressure and communication delay of vehicle-road collaboration, and improves the vehicle's perception ability and target recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the field of vehicle networking and intelligent vehicle technology, and specifically to a three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road collaboration scenario. Background Art

[0002] A single smart car cannot perceive the overall environment due to limited perception range, occlusion and other problems, and may make unsafe decisions. The vehicle-road group intelligence collaborative service system is an active traffic management service, in which the group intelligence perception subsystem needs to provide on-demand cognitive services to smart cars. The perception sources of smart cars can be divided into self-perception sources and roadside perception sources. Among them, the self-perception source is the self-perception source that the vehicle itself perceives by relying on on-board sensors, while the roadside perception source is the global perception information provided to the vehicle by the roadside collaborative service system after the fusion of multi-view and multi-directional sensors on the roadside. Smart vehicles use vehicle networking technology to interact with the roadside perception center and share the perceived environmental information, so as to collaboratively establish a complete understanding of the surrounding environment, provide accurate semantic information for subsequent planning and decision-making, positioning and other parts, and achieve safe driving tasks. However, the amount of original perception information data is huge, and the transmission delay caused by a large amount of data will bring great potential dangers to intelligent driving tasks. Cameras and lidars are common sensors in the field of smart cars. Due to the differences in perception principles, these two sensors can capture environmental information with different characteristics. Camera sensors can only capture the two-dimensional appearance information of the scene and lack depth information. Although many researchers have used monocular or binocular cameras to obtain depth information estimates, it is still difficult to meet the requirements for three-dimensional target detection accuracy in autonomous driving. LiDAR sensors can not only capture the two-dimensional information of the scene, but also include accurate depth information, which is crucial for autonomous driving perception systems. It can be seen that a single sensor source information is difficult to meet the needs of autonomous driving.

[0003] Existing document 1 (publication number CN111222441A) disclosed a point cloud target detection and blind spot target detection method and system based on vehicle-road collaboration on June 2, 2020. By cleverly changing the position of the camera and proposing a square retrieval suggestion area, it is applied to the new scene of vehicle-road collaborative perception, achieving the effect of generating better point cloud retrieval suggestion areas and target detection in blind spots, ensuring that the independence of single-vehicle intelligence is not affected in any way under street light failure, so that smart cars have faster, more comprehensive and safer target perception performance with the assistance of roadside intelligence. However, this technical solution only uses a single sensor on the roadside, does not build a rich multimodal sensor system, and does not perform roadside multimodal multi-sensor fusion with feature granularity, and cannot adapt to more complex vehicle-road collaborative scenarios.

[0004] Existing document 2 (publication number CN111222441A) disclosed a target detection and classification method based on deep learning under vehicle-road collaboration on February 11, 2020. Based on deep learning, the target is accurately detected and classified, which not only determines the location of the target, but also determines the type of the target, thus achieving effective management of urban traffic. However, this technical solution uses lidar data for three-dimensional target detection, but does not consider the influence of distance and occlusion in target detection. The detection accuracy will be affected as the distance increases and the scale of the object changes, and it cannot adapt to complex road conditions.

[0005] Existing document 3 (publication number CN111432375A) disclosed a V2X-based vehicle-road collaboration method and vehicle-road collaboration system on July 17, 2020. The second vehicle collects first data information for the first vehicle, and the V2X communication module collects second data information, fuses the data, eliminates blind spots in information transmission, and shares the fused information with the rear vehicle to collaboratively complete driving decisions. However, this technical solution only considers vehicle-to-vehicle communication and data fusion transmission, and does not consider the use of roadside sensors for vehicle-road collaboration assistance. At the same time, the data transmission part of this method shares the original data to other vehicles in real time through the network, which has high requirements for transmission rate and bandwidth, and cannot adapt to environments with poor network conditions.

[0006] Existing document 4 (publication number CN110532896A) disclosed a road vehicle detection method based on the fusion of roadside millimeter-wave radar and machine vision on December 3, 2019. This technical solution calculates the detection results of the roadside millimeter-wave radar, projects the radar detection results into the image coordinate system, and later fuses them with the target detected by the image. This solution has high requirements for the detection of the two sensors respectively, does not perform feature-level modeling, and fuses the sensors more adaptively.

[0007] At present, the autonomous driving perception system has the above problems: using only a single sensor cannot adapt to more complex vehicle-road cooperative scenarios; the detection accuracy is affected as the distance increases and the scale of the object changes, and it cannot adapt to complex road conditions; it has high requirements for transmission rate and bandwidth, and is not suitable for environments with poor network conditions; it has high requirements for two sensor detections, and cannot adaptively fuse sensor data, etc. Existing technologies cannot meet the requirements for three-dimensional target detection accuracy in autonomous driving, so further technical improvements are needed. Summary of the invention

[0008] In view of the above-mentioned problems existing in the prior art, the present invention provides a three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road cooperative scenario, which combines the sensors of the intelligent driving vehicle and the smart road. Adaptive sensor fusion and feature aggregation based on meta-fusion are first performed on the smart road part, and the fused feature data is sent to the intelligent driving vehicle through vehicle-road cooperative communication. The intelligent driving vehicle obtains the final detection result based on the multi-scale three-dimensional detection algorithm.

[0009] The present invention provides a three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road cooperative scenario, including:

[0010] Step 1: Establish a collaborative perception and shared data model in a vehicle-road collaborative scenario;

[0011] The purpose of vehicle-road cooperative perception is to provide global perception information for smart cars, so that the perception range of smart cars is not limited by the range of their own sensors, and provide an important basis for subsequent global path planning. The present invention includes roadside equipment, roadside feature fusion center and different traffic participants in the vehicle-road cooperative scenario. The roadside equipment is provided with a computing unit and a variety of sensors, which capture different environmental information through a variety of sensors, extract features from the perceived environmental information through the computing unit, and obtain multimodal features for 3D target detection. The roadside equipment transmits the acquired multimodal features and the center position in the global coordinate system to the nearest roadside feature fusion center in the network topology through a wired network in real time. The roadside feature fusion center performs fusion processing on the received multimodal features, and broadcasts the fused multimodal features and the global center position. Traffic participants within the service range of the roadside feature fusion center obtain the fused multimodal features and the global center position of the roadside feature fusion center.

[0012] Step 2: The smart car uses the feature extraction module to extract multi-source perception features from its own perception data. After receiving the multi-source fusion features sent by the roadside feature fusion center, the smart car calibrates and fuses the features with the multi-source perception features detected by itself to obtain comprehensive perception information in the current area, which is further used for target recognition and detection.

[0013] Step 3, implement the feature extraction module in the roadside equipment and the smart car. The feature extraction module includes: a point cloud feature extraction module extracts three-dimensional point cloud features from the point cloud data acquired by the sensor; and an image feature extraction module extracts two-dimensional image features from the image acquired by the sensor.

[0014] The point cloud feature extraction module adopts a neighborhood adaptive feature extraction method, which generates an adaptive neighborhood radius range for each key point through convolution, so that the features can more robustly represent the information of objects of different scales.

[0015] Step 4: Implement feature calibration and fusion modules in roadside equipment, roadside feature fusion center, and smart cars.

[0016] The feature calibration and fusion module, when fusing the point cloud feature map and the image feature map, first obtains the projection relationship matrix M of the two maps according to the calibration matrix of the point cloud feature map and the image feature map, and then performs feature level fusion based on the meta-learning method, including:

[0017] (1) Assume that the feature extraction module extracts a point cloud feature Fp from any point p sampled from the point cloud data, then the image feature F corresponding to the point p is I =f(M×p), where f is the operation of the image feature extraction module;

[0018] (2) The image feature F I The kernel parameters of the convolution kernel are generated through the fully connected layer, and a group of 1×1 convolutions are generated based on the kernel parameters to process point cloud features. The 1×1 convolution outputs a feature representation F that combines image features and point cloud features.

[0019] The feature calibration and fusion module uses the mean method or the maximum pooling method to fuse homogeneous features.

[0020] Step 5: The 3D target detection head on the smart car processes the acquired perception information and identifies the target.

[0021] Compared with the prior art, the advantages and positive effects of the present invention are:

[0022] (1) Compared with the existing technology that relies on a single sensor for three-dimensional target detection, the method of the present invention sets a variety of sensors with complementary perception features on the roadside equipment. Different sensors have different capture capabilities for environmental texture, obstacle distance, color and other features, thereby greatly improving the external environment perception of the Internet of Vehicles and smart cars, so that the designed multi-sensor joint perception achieves more robust detection results.

[0023] (2) There are many roadside devices and the original lidar data is too large. Directly sharing the original perception data will cause great communication pressure and communication delay. The present invention fuses the multimodal perception information through the roadside feature fusion center to generate a representative feature point cloud to serve traffic users and reduce the communication pressure and communication delay in vehicle-road collaboration. At the same time, using point clouds as feature carriers greatly simplifies the feature alignment problem in vehicle-road collaboration, greatly reduces the amount of data transmission compared to the original perception data, and also reduces the communication pressure and computing pressure of smart cars.

[0024] (3) Different sensors have different physical principles, so the environmental characteristics they capture are quite different. Using a single sensor in the vehicle-road collaboration framework makes it difficult to simultaneously meet the environmental texture information, color information, depth information, shape information, etc. required for intelligent vehicle perception. In the method of the present invention, a gating mechanism and an attention mechanism are used to fuse multiple perception information from cameras and lidar point clouds, thereby providing the vehicle with richer environmental information and improving the perception ability of intelligent driving vehicles. The multimodal fusion method used has strong robustness.

[0025] (4) Existing methods for feature extraction based on raw point clouds usually use a fixed-scale feature extraction method, the scale of which needs to be manually adjusted, and it is difficult to adapt to target recognition tasks with large scale differences. The method of the present invention uses a scale-adaptive local feature extraction method, so that the network can adapt to different target sizes and has good robustness. The feature extraction module of the present invention extracts both three-dimensional point cloud features and two-dimensional image features to obtain more comprehensive environmental features, thereby improving the accuracy of target recognition in the environment.

[0026] (5) Compared with the existing technology that uses indiscriminate detection for objects that are near, far, and have different degrees of occlusion, the method of the present invention takes into account that the point cloud density of distant objects captured by the lidar sensor is significantly sparser than that of nearby objects, and that objects with a high degree of occlusion have different features from objects with a low degree of occlusion. The designed scale-adaptive three-dimensional target detection head can generate an adaptive feature extraction range for the above different targets, thereby making the method of the present invention adaptable to more complex vehicle-road collaborative scenarios. Compared with the existing technology that only considers cooperative perception between vehicles, the method of the present invention uses roadside sensors to assist in vehicle-road collaboration, and adopts a feature data collaborative assistance strategy. Compared with the strategy of transmitting raw data collaboration, it is more suitable for the field of intelligent transportation with higher communication delay requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road cooperative scenario of the present invention;

[0028] Figure 2 It is a schematic diagram of the implementation of the three-dimensional target detection and recognition method based on multi-source fusion in the vehicle-road cooperative scenario of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0030] The three-dimensional target detection and recognition method based on multi-source fusion in the vehicle-road cooperative scenario implemented by the present invention mainly includes the following five parts, which correspond to the following respectively: Figure 1The five steps shown are explained.

[0031] In the first part, considering the usage of sensor resources and computing resources of smart cars and combining the characteristics of vehicle sensors, a multi-source mid-term fusion method is designed. The roadside feature fusion center provides perception information to smart cars in the form of intermediate features, which greatly reduces the data volume and delay of the original perception information transmission.

[0032] In the second part, as the main traffic participant, smart cars not only receive the multi-source fusion features sent by the roadside feature fusion center, but also obtain multi-source perception features themselves, calibrate and fuse the two to obtain more comprehensive environmental perception information.

[0033] In the third part, the present invention extracts three-dimensional point cloud features and two-dimensional image features from environmental information to ensure that environmental information is fully acquired. In the intelligent traffic scenario, target detection is performed, and the detected target objects are usually cars, pedestrians, cyclists, etc. The scales of objects of different categories vary greatly, so it is usually necessary to consider the problem of inconsistent scales of multiple target objects. Generally, researchers rely on multi-scale feature fusion to improve the generalization of object detection with inconsistent scales, but extracting multi-scale features increases the amount of calculation. Considering that the objects of interest in the smart road scenario have different scales, the present invention proposes a neighborhood adaptive feature extraction method. The feature extraction module generates an adaptive neighborhood radius at each key point by convolution to convert the fixed-scale spherical neighborhood into an adaptive neighborhood, and supervises the generated spherical neighborhood radius by trilinear interpolation method, so that the network obtains feature information of different scales adapted to objects of different categories, so that the features can more robustly represent the information of objects of different scales.

[0034] In the fourth part, the present invention takes into account the irregularity and disorder of lidar data, as well as the regularity and orderliness of image data, takes into account the characteristics of the two types of data, jointly analyzes the different levels of geometric and semantic information contained in the multimodal data, and designs a multimodal fusion method based on gating mechanism and attention mechanism, which provides input information for vehicle perception more efficiently and improves the perception ability of intelligent driving vehicles.

[0035] In the fifth part, the smart car performs target detection based on the information obtained from the current scene.

[0036] The specific implementation of each step of the three-dimensional object detection and recognition method of the present invention is described below.

[0037] The first step is to explain the implementation framework of target detection based on multi-source fusion under vehicle-road collaboration implemented by the present invention.

[0038] To achieve beyond-visual-range scene recognition, smart cars not only need to rely on the environmental information captured by their own sensors for 3D target detection, but also need environmental perception information provided by the roadside. The overall technical solution structure for beyond-visual-range scene recognition under vehicle-road collaboration is as follows: Figure 2 As shown. Figure 2 As shown, in the vehicle-road collaboration scenario of the present invention, in addition to the roadside equipment and the smart car, a roadside feature fusion center is also provided. The roadside feature fusion center can be installed at an appropriate distance according to the location of the roadside equipment.

[0039] The set of sensing units in the roadside equipment can be expressed as O Infra ={S1,S2,...,S n}, n represents the number of perception units. Each infrastructure unit contains multiple sensor devices, and different sensors can capture different environmental information; in order to perform multi-source fusion feature extraction, multiple sensors must be calibrated in advance to obtain the corresponding rotation matrix R and translation matrix P. The computing unit in the roadside equipment is a lightweight computing unit for extracting 3D target detection features. The computing unit is provided with a multi-modal feature extraction module, and a feature calibration and fusion module. The perception data captured by the roadside perception unit set and the calibration matrix (centered on the camera sensor) are input into the computing unit to extract multimodal features for 3D target detection. In the computing unit of the roadside equipment, the three-dimensional point cloud features and two-dimensional image features are obtained through the feature extraction module, and the features captured by the sensor are fused through the feature calibration and fusion module.

[0040] After each roadside computing unit obtains the multimodal features, it is necessary to fuse the multimodal features between the roadside computing units at the roadside feature fusion center. The characteristics of multimodal feature fusion between different roadside sensor groups are that the features of multiple groups of sensors are equally important, and there is a certain correlation between the features. In the method of the present invention, when the multimodal features transmitted from different roadsides are fused using the feature calibration and fusion module, the Transformer model is used to calculate the weights of the two groups of feature vectors, and the feature vectors and their cross product vectors are spliced ​​and sent to the fully connected layer to calculate the weights. By splicing the cross product vectors, the correlation information is explicitly introduced to assist the calculation of the attention mechanism. Finally, the unified features after the fusion of the roadside sensor groups are obtained.

[0041] The roadside infrastructure unit transmits the multimodal features and the center position in the global coordinate system to the nearest roadside feature fusion center in the topology through a wired network. The roadside feature fusion center is equipped with a feature calibration and fusion module, which fuses the multimodal features received from multiple roadside devices, outputs the fused roadside fusion multimodal features and the global center position of the roadside feature fusion center, and provides perception services to each traffic participant within the service range through wireless communication. Each roadside feature fusion center can provide services to traffic agents within a certain service coverage area. When a traffic agent enters the service range, the roadside feature fusion center can actively perceive and transmit real-time roadside fusion multimodal features to it.

[0042] In the second step, after receiving the multi-source fusion features sent by the roadside feature fusion center, the smart car calibrates and fuses the features with the multi-source perception features detected by itself to obtain comprehensive perception information in the current area.

[0043] Smart cars are intelligent entities with wireless information receiving devices, their own sensors and computing units, and are the main traffic participants. Smart cars can initially recognize the surrounding environment through their own set of perception units. The aligned perception data is first input into the feature extraction module to extract the self-multi-source perception features. In order to achieve beyond-visual-range cognition, smart cars can receive multi-source fusion features transmitted by the roadside feature fusion center in the service area through wireless networks. Smart cars process the multi-source fusion features and self-multi-source perception features through feature calibration and fusion modules to obtain comprehensive perception information in this area. Finally, the multi-source features are input into the detection head to obtain the semantic information of the target in this area, including the target category, absolute coordinate position and target size.

[0044] The third step is to implement the feature extraction module in the roadside equipment and the smart car. The functions implemented by the feature extraction module of the present invention mainly include two parts: one is the point cloud feature extraction module, which is responsible for extracting three-dimensional point cloud features; the other is the image feature extraction module, which is responsible for extracting two-dimensional image features.

[0045] (3.1) For the extraction of point cloud features, the point cloud feature extraction module of the present invention abstracts point cloud features by sampling key points and aggregating neighborhood point features. The neighborhood query method used is spherical neighborhood query, and the adaptive spherical neighborhood radius can achieve the purpose of scale adaptation. Therefore, the present invention proposes a neighborhood radius generation module to extract scale information. In the neighborhood radius generation module, the radius value generated is used to adaptively aggregate neighborhood point features, and the neighborhood radius value is back-propagated and iteratively updated through additional supervision tasks.

[0046] When the point cloud feature extraction module extracts point cloud features, it can be divided into three steps: (1) The point cloud frame is voxelized, and then the voxelized data is input into the convolutional neural network to extract the shape information of the potential target object of each voxel block, and finally input into the detection head composed of a fully connected network of the neighborhood radius generation module to generate the neighborhood radius value corresponding to this voxel block. (2) The farthest point sampling method is used to sample the global key points of the point cloud frame, and it is projected to the neighborhood radius grid obtained in (1) to obtain an adaptive radius value. (3) For each key point, N neighborhood points are sampled in the neighborhood radius obtained in (2), and the N neighborhood point features are aggregated by the maxPooling method to obtain the key point feature. N is a positive integer, and N is selected as 16 in the embodiment of the present invention.

[0047] The neighborhood radius generation module of the present invention adopts a feature extraction method similar to that in the three-dimensional sparse convolution module. The difference is that in order to avoid the loss of scale information, only two sets of sparse convolution layers SparseConvNet are used to extract the scale information related features of the potential target at the voxel, and the number of output parameters of the last set of sparse convolution layers is modified to 1 to directly generate the neighborhood radius value of the neighborhood point relative to the center position of the voxel. Finally, for the voxel V i , will produce a radius R i ; Voxel V i The 16 neighborhood points can be located in voxel V i Centered, R i It is generated by random sampling within the radius, represented as a set

[0048] The feature calculation of key points is usually obtained by aggregating the features of 16 randomly sampled neighborhood points within the neighborhood radius using the maxPooling method.

[0049] At the same time, in order to realize the supervised learning of the neighborhood radius generation module, the present invention adds an additional supervision task L for the neighborhood cube R-box For voxel V i The neighborhood cube is defined as R generated by the neighborhood radius generation module as the side length and voxel V i The center point position is used as the center coordinate of the cube, that is Represents voxel V i The center point coordinates, Represents the length, width and height of the cube.

[0050] Supervision Task L shifting-box Use smooth L1 Loss for calculation, as shown in the following formula:

[0051]

[0052] Among them, Cgt ={x gt ,y gt ,z gt ,l gt ,w gt ,h gt}, is the real frame value.

[0053] (3.2) For the extraction and fusion of two-dimensional image features, the image feature extraction module of the method of the present invention adopts Faster-RCNN as the image feature extraction architecture, wherein the feature extraction backbone network adopts the residual network architecture. The image feature extraction architecture is pre-trained in advance using a large image dataset to ensure the image extraction effect.

[0054] The fourth step is to implement the feature calibration and fusion modules in the roadside equipment, roadside feature fusion center and smart car. The feature calibration and fusion module can be divided into a calibration module and a feature fusion module. The calibration module mainly realizes the conversion of features under different sensor coordinates, and the feature fusion feature mainly fuses homogeneous or heterogeneous features. In the application scenario of the present invention, heterogeneous features refer to three-dimensional point cloud features and two-dimensional image features. Homogeneous features can be multi-modal features after fusion, or three-dimensional point cloud features, etc.

[0055] For point cloud feature maps and image feature maps with calibration matrices, the feature calibration and fusion module can obtain the projection relationship between the point cloud and the image, and perform projection fusion through the projection relationship. The present invention proposes a feature level fusion method based on a meta-learning method, also known as a meta-fusion method. For any point p sampled from the point cloud P, the point cloud feature Fp is obtained through the point cloud feature extraction module, and the image feature F of the point p can be obtained through the projection matrix M. I for:

[0056] F I =f(M×p)

[0057] Where f is the image feature extraction module. The image features and point cloud features are used to interact to fuse the point cloud features and image features, and the final feature representation F is obtained.

[0058] In the fusion process, the present invention uses 1×1 convolution to implement the meta-fusion method. The meta-fusion method transforms the image feature F I The kernel parameters of the convolution kernel are then generated through the fully connected layer, and then a group of 1×1 convolutions generated by image features are used to process the point cloud features, and the number of convolution kernels is used to control the dimension of the final output feature. The meta-fusion method combines point cloud features and image features to obtain an adaptive multi-modal feature representation as the input of subsequent modules.

[0059] Since each sensor uses its own center as the data origin in its physical configuration, it is necessary to convert the sensor data to the global coordinate system in order to perform multi-source fusion recognition operations. The present invention uses a feature calibration and fusion module to complete this function. The data from different sensors are aligned by rotation and translation operations. The points are mapped from the sensor coordinate system to the global coordinate system and specified by the inverse of the external matrix of each sensor. The external matrix is ​​the calibration matrix. Given the coordinates (x, y, z) of a point in the coordinate system of sensor i, the global coordinates (x g ,y g ,z g ):

[0060]

[0061] Among them, M i is the external matrix of sensor i, which can be decomposed into the rotation matrix R i With the translation vector t i The sensor’s rotation matrix and translation vector must be obtained through a calibration process. This is challenging in practice because the external matrix M i depends on the position and orientation of the sensors, therefore, the result can only be as accurate as the measured values ​​of these variables. In practice, if these sensors are mounted on mobile nodes, such as vehicles, any errors in the positioning of the mobile nodes will lead to alignment errors in the fused point cloud, which may result in false positives and missed detections. In the model of the application scenario of the present invention, the sensors are fixed to the roadside; therefore, the calibration process can be performed very accurately in practice. Through the feature calibration step, the external calibration matrix M between the two fusion modules can be obtained i Through rotation and translation operations, the features of perception unit A can be transformed into the coordinate system of perception unit B, and the transformed feature point cloud of perception unit A can be fused with the feature point cloud of perception unit B to obtain the fused features.

[0062] The homogeneous features of the present invention mainly refer to multimodal features. The Transformer model is first used to calculate the weights of two sets of input features, and then the maxPooling (maximum pooling) or mean method is used for fusion.

[0063] The fifth step is to implement the 3D object detection head on the smart car.

[0064] After feature extraction is completed and calibration fusion is performed, the network inputs the fused features into the 3D sparse convolution module, which receives the voxelized point cloud information and extracts the high-level features of the point cloud through four sets of sparse convolution layers. Since the point cloud itself is very sparse, most of the generated voxelized 3D grids do not have valid feature information. In order to reduce storage resource consumption, the voxelized point cloud information is stored in a sparse matrix, that is, the 3D voxel grid is represented as {V1, V2, V3, ..., V n}, where V i represents the i-th non-empty voxel, which is represented in the form of a sparse matrix, that is, in is a non-empty voxel V i The voxel center position coordinates, is a non-empty voxel V i Feature values ​​extracted by PointNet network model. PointNet is a SparseConvNet module (sparse convolution module) currently commonly used in 3D object detection. The method of the present invention uses four groups of SparseConvNet to extract point cloud 3D features from the currently obtained regional comprehensive perception information, where each group of SparseConvNet consists of a convolution layer with a step size of 2 and two sparse convolutions with a step size of 1 to reduce the size of the feature map and extract sufficiently abstract feature information.

[0065] After obtaining high-level features, the high-level features are compressed and mapped to the BEV plane along the height, and the initial detection box is obtained through the region proposal network.

[0066] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of the known technologies to avoid redundant description and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.

Claims

1. A three-dimensional target detection and recognition method based on multi-source fusion in a vehicle-road cooperative scenario, characterized in that: The steps include: Step 1: The vehicle-road collaboration scenario includes roadside equipment, roadside feature fusion center, and different traffic participants; Among them, a variety of sensors are set in the roadside equipment to capture environmental information, and then features are extracted from the various environmental information through the feature extraction module, and the image features and point cloud features are fused by the feature calibration and fusion module to obtain multimodal features for 3D target detection; the roadside equipment transmits the calculated multimodal features and its own global center position to the nearest roadside feature fusion center in the network topology through a wired network; the roadside feature fusion center uses the feature calibration and fusion module to fuse the multimodal features sent from each roadside equipment to obtain multi-source fusion features, and broadcasts the multi-source fusion features and its own global center position; traffic participants within the service range of the roadside feature fusion center obtain the multi-source fusion features and the global center position of the roadside feature fusion center; Step 2: The smart car uses the feature extraction module to extract multi-source perception features from its own perception data. After receiving the multi-source fusion features sent by the roadside feature fusion center, it uses the feature calibration and fusion module to fuse them with the multi-source perception features detected by itself to obtain comprehensive perception information in the current area. Step 3, realizing a feature extraction module in the roadside equipment and the smart car; the feature extraction module includes a point cloud feature extraction module and an image feature extraction module; the point cloud feature extraction module extracts three-dimensional point cloud features from the point cloud data acquired by the sensor; the image feature extraction module extracts two-dimensional image features from the image acquired by the sensor; Step 4, realizing feature calibration and fusion modules in roadside equipment, roadside feature fusion center and smart car; the feature calibration and fusion module, for the fusion of point cloud feature map and image feature map, first obtains the projection relationship matrix M of the two maps according to the calibration matrix of the point cloud feature map and the image feature map, and then performs feature hierarchical fusion based on the meta-learning method, including: (1) Assume that the feature extraction module extracts a point cloud feature Fp from any point p sampled from the point cloud data, then the image feature F corresponding to the point p is I =f(M×p), where f is the operation of the image feature extraction module; (2) The image feature F I Generate kernel parameters of the convolution kernel through the fully connected layer, and generate a group of 1×1 convolutions based on the kernel parameters to process point cloud features. The 1×1 convolution outputs a feature representation F that combines image features and point cloud features. Step 5: The smart car detects 3D targets based on the comprehensive perception information of the current area.

2. The method according to claim 1, characterized in that In step 3, when the point cloud feature extraction module extracts point cloud features, it can be divided into three steps: (1) voxelizing the point cloud frame, and then inputting the voxelized data into a convolutional neural network to extract the shape information of the potential target object of each voxel block and generate a neighborhood radius value corresponding to the voxel block; (2) The global key points of the point cloud frame are sampled by the farthest point sampling method, and the global key points are projected onto the neighborhood radius grid obtained in (1) to obtain an adaptive radius value; (3) For each key point, sample N neighboring points in the neighborhood radius obtained by (2), and aggregate the features of the N neighboring points through the maxPooling method to obtain the key point features; N is a positive integer.

3. The method according to claim 2, characterized in that In the step 3, a neighborhood radius generation module is set in the point cloud feature extraction module. The neighborhood radius generation module only uses two groups of sparse convolutional layers to extract relevant features of the shape scale information of the voxel potential target, and modifies the number of output parameters of the last group of sparse convolutional layers to 1, directly generating the neighborhood radius value of the neighborhood point relative to the voxel center position.

4. The method according to claim 1, characterized in that In the step 4, each sensor provided in the roadside equipment is pre-calibrated to obtain the corresponding rotation matrix and translation matrix; the feature calibration and fusion module uses the rotation matrix and translation matrix of the sensor to perform coordinate transformation on the features under different sensor coordinates.

5. The method according to claim 1 or 3, characterized in that: In step 4, the feature calibration and fusion module uses the Transformer model to calculate the weights of the two sets of input features for the fusion of homogeneous features, and then uses the maximum pooling method to perform feature fusion.

6. The method according to claim 1 or 3, characterized in that: In step 4, the feature calibration and fusion module uses the Transformer model to calculate the weights of the two sets of input features for the fusion of homogeneous features, and then uses the mean method to perform feature fusion.

7. The method according to claim 1 or 3, characterized in that: In the step 5, when detecting 3D targets, four groups of sparse convolutional layers are used to extract high-level features from the currently obtained regional comprehensive perception information, and each group of sparse convolutional layers consists of a convolutional layer with a step size of 2 and two sparse convolutions with a step size of 1; the obtained high-level features are compressed and mapped to the BEV plane along the height, and the initial detection box is obtained through the region proposal network.

Images