Target detection method based on fusion of millimeter wave radar and visual features

Abstract

The application discloses a target detection method based on fusion of millimeter wave radar and visual features, comprising the following steps: acquiring a calibrated millimeter wave radar point cloud and a camera image, projecting the millimeter wave radar point cloud to an image pixel plane through three-dimensional space coordinate transformation; performing plane expansion processing in the vertical and horizontal directions on the millimeter wave radar point cloud projected to the image pixel plane; extracting features of the millimeter wave radar point cloud to obtain a radar feature map; constructing an image feature extraction network, extracting features of the camera image to obtain an image feature map; calculating the similarity of the radar feature map and the similarity of the image feature map to obtain a radar feature similarity map and an image feature similarity map; performing pixel addition on the radar feature similarity map and the image similarity map to obtain a fusion similarity map; cascading the fusion similarity map and the image feature map to obtain a fusion feature map, and cascading the sampled fusion feature map and the radar feature map; and finally obtaining a target detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular, especially relates to a target detection method based on millimeter wave radar and visual feature fusion for improving detection capability. BACKGROUND

[0002] In the field of perception, target detection for bad weather such as rain, fog, snow and occlusion problems has always been a hot issue in automatic driving target detection, and sensor fusion technology can effectively solve the problem of small target and bad weather detection loss. Among them, the radar and visual sensor fusion scheme is a relatively typical target detection scheme, which realizes complementation by using the advantages of each target detection, and the radar can realize accurate positioning of the target, is less affected by light and weather interference, the visual obtains rich image information, and has high resolution, which is beneficial to the extraction of target deep semantics.

[0003] The existing technical scheme divides the fusion of vision and millimeter wave radar into data level, target level and feature level. Feature level fusion is a fusion method between data level fusion and decision level fusion. In recent years, with the development of deep learning, feature level fusion has developed rapidly due to its rich potential to utilize radar point cloud speed, distance and other different modal information.

[0004] Feature level fusion is to fuse radar point cloud feature maps in a deep learning network, which can make full use of the multi-modal information contained in the point cloud and improve the detection capability. However, at the same time, the point cloud itself has the natural sparseness, especially the number of filtered point clouds is much less than that of images, and there is a lack of technology for enhancing the semantic information of radar features, and the noise of radar will also interfere with the detection result. Therefore, how to efficiently fuse sparse radar feature maps is an important measure to improve the detection effect. In addition, the construction and use of the detection network need to consider the efficiency of calculation and the accuracy and reliability of prediction, and most of the existing researches directly input the radar feature maps into the visual detection network, lacking research on the detection network for fused radar features. SUMMARY

[0005] The present application aims to overcome the shortcomings and deficiencies of the prior art, and provides a target detection method based on millimeter wave radar and visual feature fusion, which improves the detection capability of small targets in bad weather under multi-modal input.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] A target detection method based on millimeter wave radar and visual feature fusion, comprising the following steps:

[0008] The calibrated millimeter wave radar point cloud and the camera image are acquired, and the millimeter wave radar point cloud is projected to an image pixel plane through three-dimensional space coordinate transformation;

[0009] The millimeter wave radar point cloud projected to the image pixel plane is subjected to plane expansion processing in the vertical and horizontal directions;

[0010] A radar feature extraction network is constructed to extract features of the millimeter wave radar point cloud, and a radar feature map is obtained;

[0011] An image feature extraction network is constructed to extract features of the camera image, and an image feature map is obtained;

[0012] A similarity calculation module is constructed to calculate the similarity of the radar feature map and the similarity of the image feature map, and a radar feature similarity map and an image feature similarity map are obtained;

[0013] A similarity fusion module is constructed to add the radar feature similarity map and the image similarity map pixel by pixel to obtain a fused similarity map;

[0014] A feature pyramid network is constructed to cascade the fused similarity map and the image feature map to obtain a fused feature map, and the fused feature map is sampled and cascaded with the radar feature map;

[0015] A target classification regression subnetwork is constructed to obtain a target detection result, and the size, position and classification result of the detected target are output.

[0016] Preferably, the millimeter wave radar point cloud is projected to the image pixel plane through three-dimensional space coordinate transformation, specifically:

[0017] The millimeter wave radar point cloud is converted from a radar coordinate system coordinate to a world coordinate system coordinate centered on the camera;

[0018] The conversion model is:

[0019]

[0020] In the formula, X W , Y W , and Z W are world coordinate system coordinates, X R , Y R , and Z R are radar coordinate system coordinates, H RW is the distance of the radar coordinate system coordinate Y R relative to the world coordinate system coordinate Y W , and Z RW is the distance of the radar coordinate system coordinate Z R relative to the world coordinate system coordinate Z W ;

[0021] Convert the millimeter wave radar point cloud world coordinate system coordinates through the camera coordinate system and the image coordinate system to the pixel coordinate system;

[0022] The conversion model is:

[0023]

[0024] In the formula, M1 and M2 are camera internal and external parameter matrices, u and v are pixel coordinate system coordinates, Z C is the camera coordinate system coordinates.

[0025] Preferably, the millimeter wave radar point cloud projected onto the image pixel plane is subjected to plane expansion in the vertical and horizontal directions, specifically:

[0026] The projected radar points are subjected to vertical pillar expansion, distance, reflectivity and radial relative speed are selected as three-channel information, a three-channel blank feature map with the same size as the camera image is generated, the distance, reflectivity and radial relative speed contained in the projected radar points are filled into the corresponding position points of the blank feature map, and the radar points with a given height above the point are projected onto the blank feature map plane, and the point cloud vertical expansion line is obtained by connecting and filling the pixel points between the two points.

[0027] The projected radar points are subjected to horizontal uncertainty expansion, for the blank feature map after vertical expansion, it is assumed that the measurement of the azimuth angle of each radar point is a Gaussian distribution, the measured azimuth angle is used as the mean value, and the accuracy in the radar manual is used as the standard deviation, the vertical expansion line of each point is horizontally expanded according to the Gaussian distribution, and the maximum value is used in the case of radar point expansion overlap.

[0028] Preferably, a radar feature extraction network is constructed to extract the features of the millimeter wave radar point cloud to obtain a radar feature map, specifically:

[0029] For each radar point, distance, reflectivity and radial relative speed are selected as three-channel radar features;

[0030] The three-channel radar feature map is adjusted through a 1*1 size convolution kernel convolution operation;

[0031] The three-channel radar feature map is down-sampled to the same size as the corresponding scale image feature map through equivalent convolution operation.

[0032] Preferably, the radar feature extraction network adopts a parallel seven-branch structure, the first five branches are used to down-sample the radar feature map to the same size as the corresponding scale image feature map, and the last two branches are used to down-sample the radar feature map for a feature pyramid network, each branch includes a 1*1 convolution layer and an equivalent convolution which adjusts the radar feature map to the same size as the corresponding scale image feature map.

[0033] Preferably, an image feature extraction network is constructed to extract features of the camera image to obtain an image feature map, specifically:

[0034] A convolutional neural network VGG16 is used as the image feature extraction network, and the VGG16 includes 5 blocks, 13 convolutional layers and 3 fully connected layers; the camera RGB image is input into the VGG16, and the VGG16 outputs different scale image feature maps.

[0035] Preferably, a similarity calculation module is constructed to calculate the similarity of the radar feature map and the similarity of the image feature map to obtain a radar feature similarity map and an image feature similarity map, specifically:

[0036] For each channel in each scale radar feature map, an adaptive max-pooling operation is performed to compress the radar feature map into a radar feature block;

[0037] For each channel in each scale image feature map, an adaptive average-pooling operation is performed to compress the image feature map into an image feature block;

[0038] The radar feature map and the radar feature block, and the image feature map and the image feature block are respectively calculated for similarity to obtain different feature map-point block similarity maps;

[0039] The similarity calculation formula is:

[0040]

[0041] In the formula, x i is any point of the radar feature map or the image feature map, x j is any point of the radar feature block or the image feature block, f(x i ) is the size of any point of the radar feature map or the image feature map, f(x j ) is the size of any point of the radar feature block or the image feature block, and Sim represents the similarity.

[0042] Preferably, a similarity fusion module is constructed to pixel-wise add the radar feature similarity map and the image similarity map to obtain a fused similarity map, specifically:

[0043] The radar feature similarity maps of different channels of each scale are concatenated to generate a radar feature similarity map;

[0044] The radar feature similarity map and the image feature similarity map are pixel-wise added, and then the channel number is adjusted through a 1*1 convolutional layer to generate a fused similarity map;

[0045] The fused similarity map and the different scale image feature maps generated in the VGG16 are concatenated to generate different scale fused feature maps.

[0046] Preferably, a feature pyramid network is constructed, and the similarity fusion map is cascaded with the image feature map to obtain a fusion feature map, and the fusion feature map after sampling is cascaded with the radar feature map, specifically:

[0047] Starting from the smallest scale, the fusion feature maps of different scales are twice up-sampled through the feature pyramid network, and the up-sampling mode adopts transposed convolution;

[0048] The fusion feature map of the smallest scale and the fusion feature map after each step of up-sampling are cascaded with the radar feature map of the same scale, and output to the target classification regression sub-network;

[0049] The fusion feature map of the smallest scale is twice down-sampled, and the fusion feature map after twice down-sampling is cascaded with the radar feature map of the same scale, and output to the target classification regression sub-network.

[0050] Preferably, a target classification regression sub-network is constructed to obtain a target detection result, and the size, position and classification result of the detected target are output, specifically:

[0051] Each scale fusion feature map is passed through a target classification sub-network and a convolution with a size of the number of prior boxes multiplied by the number of classification categories, and the prediction of each prior box for different categories is output;

[0052] Each scale fusion feature map is passed through a target regression sub-network and a convolution with a size of the number of prior boxes multiplied by 4, and the adjustment of each prior box is output;

[0053] The target classification sub-network is 4 times of 256-channel convolution, and the target regression sub-network is 4 times of 256-channel convolution.

[0054] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0055] 1. The present application aims at the problem of inconsistency between millimeter wave radar and image two modalities, adopts a similarity fusion method, unifies the two modalities, introduces the feature layer containing radar and image semantic information into the target detection network through the similarity fusion module, effectively utilizes the detection ability of millimeter wave radar for small targets in bad weather, and improves the detection effect.

[0056] 2. The method of millimeter wave radar point cloud plane expansion is used to simultaneously expand the sparse millimeter wave radar point cloud in the horizontal and vertical directions, improve the sparsity of the radar point cloud, and increase the semantic information contained in the feature layer.

[0057] 3. A lightweight millimeter wave radar feature extraction network is designed to improve the problem that the traditional serial network will continuously reduce the semantic information contained in the radar point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flow chart of a target detection method based on fusion of millimeter wave radar and visual features.

[0059] Figure 2 A network structure diagram in a target detection method based on fusion of millimeter wave radar and visual features.

[0060] Figure 3 An S&F module structure diagram in a similarity calculation and fusion network. DETAILED DESCRIPTION

[0061] The target detection method based on fusion of millimeter wave radar and visual features will be further described below in combination with the drawings and specific embodiments.

[0062] Referring to Figure 1 and Figure 2 , the present application discloses a target detection method based on fusion of millimeter wave radar and visual features, comprising the following steps:

[0063] S1: obtaining calibrated millimeter wave radar point cloud and camera image, projecting the millimeter wave radar point cloud to the image pixel plane through three-dimensional space coordinate transformation;

[0064] S2: performing plane expansion processing in the vertical and horizontal directions on the millimeter wave radar point cloud projected to the image pixel plane;

[0065] S3: constructing a radar feature extraction network, extracting features of the millimeter wave radar point cloud to obtain a radar feature map;

[0066] S4: constructing an image feature extraction network, extracting features of the camera image to obtain an image feature map;

[0067] S5: constructing a similarity calculation module, calculating the similarity of the radar feature map and the similarity of the image feature map to obtain a radar feature similarity map and an image feature similarity map;

[0068] S6: constructing a similarity fusion module, performing pixel addition on the radar feature similarity map and the image similarity map to obtain a fusion similarity map;

[0069] S7: constructing a feature pyramid network, cascading the fusion similarity map and the image feature map to obtain a fusion feature map, and cascading the sampled fusion feature map and the radar feature map;

[0070] S8: constructing a target classification regression sub-network to obtain a target detection result, and outputting the size, position and classification result of the detected target.

[0071] The application is based on a target detection method of millimeter wave radar and visual feature fusion, and a special lightweight radar feature extraction network is designed for millimeter wave radar point cloud feature extraction. The millimeter wave radar point cloud features and image features are fused, and the millimeter wave radar point cloud features are used as a supplement to the image features. The semantic information contained in the sparse point cloud is increased by expanding the millimeter wave radar point cloud in the plane. Through the similarity fusion operation of the millimeter wave radar point cloud and the image features, the two modalities are aligned, and the detection ability of the target detection to small targets in bad weather under the condition of multi-modal input is improved.

[0072] In step S1, the millimeter wave radar point cloud is projected to the image pixel plane through three-dimensional space coordinate transformation, including:

[0073] The millimeter wave radar point cloud is converted from the radar coordinate system coordinates to the camera-centered world coordinate system, and the conversion model is:

[0074]

[0075] In the formula, X W , Y W , and Z W are world coordinate system coordinates, X R , Y R , and Z R are radar coordinate system coordinates, H RW is the distance of the radar coordinate system coordinate Y R to the world coordinate system coordinate Y W , and Z RW is the distance of the radar coordinate system coordinate Z R to the world coordinate system coordinate Z W .

[0076] The millimeter wave radar point cloud world coordinate system coordinates are converted through the camera coordinate system and the image coordinate system, and finally converted to the pixel coordinate system, and the conversion model is:

[0077]

[0078] In the formula, M1 and M2 are camera internal and external parameter matrices, u and v are pixel coordinate system coordinates, and Z C is a camera coordinate system coordinate.

[0079] Step S2: The millimeter wave radar point cloud projected to the image pixel plane is processed by vertical and horizontal plane expansion.

[0080] Specifically, the projected radar points are vertically expanded, three output quantities of radar distance, reflectivity and radial relative speed are selected as three channel information, a blank three-channel radar feature map with the same size as the image is generated, and the distance, reflectivity and radial relative speed values contained in the projected radar points are filled into the corresponding position point size of the blank feature map, and the radar points with a given height (preset as 3m) above the point are projected onto the blank feature map plane to represent the uncertainty of height, and the point cloud vertical expansion line is obtained by connecting and filling the pixels between two points. The values of all points on the line are the same.

[0081] The projected radar points are horizontally expanded, and for the vertically expanded radar feature map, it is assumed that the measured azimuth angle of each radar point is a Gaussian distribution, the measured azimuth angle is used as the mean value, and the precision in the radar manual is used as the standard deviation. Each point's vertical expansion line is horizontally expanded to represent the horizontal uncertainty, and the maximum value is used in the case of radar point expansion coincidence.

[0082] Step S3: Construct a radar feature extraction network to extract the features of the millimeter wave radar point cloud to obtain a radar feature map.

[0083] Specifically, the radar feature extraction network is as shown in Figure 2 For the radar data (Radardata) of each radar point, the distance, reflectivity and radial relative speed are selected as three-channel radar features, and the radar feature extraction network is constructed. The radar feature extraction network adopts a parallel branch structure.

[0084] Each branch block first passes through a 1*1 convolution layer, and then passes through an equivalent convolution to adjust the radar feature map to the same size as the corresponding scale image feature map. The radar feature map is adjusted to the same size as the image feature map.

[0085] The radar feature extraction network adopts a parallel seven-branch block structure. The first five branches are used to downsample the radar feature map to the same size as the corresponding scale image feature map, and the last two branches are used for FPN (Feature Pyramid Networks) feature pyramid network. In this way, the loss of radar semantic information can be reduced, and the radar feature information can be maximally preserved.

[0086] Step S4: Construct an image feature extraction network to extract the features of the camera image to obtain an image feature map.

[0087] Specifically, VGG16 is used as the image feature extraction network, and the image feature extraction network is as shown in Figure 2As shown, VGG16 includes 5 blocks, a total of 16 layers, 13 convolutional layers and 3 fully connected layers, and the last three fully connected layers are removed. After two convolutions with 64 convolution kernels in the first block, one maximum pooling is used, and after two convolutions with 128 convolution kernels in the second block, maximum pooling is used again, and two same blocks are repeated, each block including three 512 convolution kernels and one maximum pooling. For image data of a camera, RGB three-channel images are input into a pre-trained network, and different scale feature maps output after each block is saved.

[0088] In VGG16, multiple small convolution kernels (3x3) are used in the convolution layer instead of one large convolution kernel, which can reduce the parameters on the one hand, and can increase the fitting or expression ability of the network by performing more nonlinear mapping on the other hand.

[0089] Step S5: Construct a similarity calculation module to calculate the similarity of the radar feature map and the similarity of the image feature map, and obtain a radar feature similarity map and an image feature similarity map,

[0090] Specifically, the similarity calculation module is as shown in Figure 3 For each channel in each scale radar feature map, an adaptive max pooling operation is performed to compress the feature map into a feature block.

[0091] For each channel in each scale image feature map, an adaptive average pooling operation is performed to compress the feature map into a feature block.

[0092] The similarity of each radar feature map and the compressed feature is calculated, and the same operation is performed on the image feature map to obtain a block similarity map of different feature map points.

[0093] The calculation formula of the similarity is:

[0094]

[0095] In the formula, x i is any point of the radar feature map or the image feature map, x j is any point of the radar feature block or the image feature block, f(x i ) is the size of any point of the radar feature map or the image feature map, f(x j ) is the size of any point of the radar feature block or the image feature block, and Sim represents the similarity.

[0096] Step S6: a similarity fusion module is constructed to perform pixel-wise addition on the radar feature similarity map and the image similarity map to obtain a fused similarity map.

[0097] Specifically, as shown in the similarity fusion module Figure 3 concatenates the radar feature similarity maps at each scale, i.e., seven C*H*W radar feature similarity maps are generated, H and W represent the width and height of the radar feature similarity map at the corresponding scale, and C is the number of channels.

[0098] The generated radar feature similarity map is subjected to pixel-wise addition operation with the corresponding image feature similarity map, specifically, the values at the corresponding positions of each pixel are added, and then a 1*1 convolution layer is used to adjust the number of channels back to three channels, i.e., the channels of each type of feature are compressed, and a fused similarity map is generated.

[0099] The generated fused similarity map is concatenated with the different scale image feature maps generated in the VGG16 to generate different scale fused feature maps, and the additional similarity channel saves the similarity information of the radar and the image in the same modality.

[0100] Step S7: a feature pyramid network is constructed to concatenate the fused similarity map and the image feature map to obtain a fused feature map, and the fused feature map is sampled and concatenated with the radar feature map.

[0101] Specifically, as shown in the feature pyramid network Figure 2 , starting from the smallest scale, the generated fused feature maps at different scales are named P5, and are upsampled to P4 and P3 through the FPN feature pyramid network, and the upsampled manner adopts transposed convolution.

[0102] The smallest scale fused feature map and the fused feature map after each step of upsampling are concatenated with the radar feature map at the same scale, and are output to the target classification and regression subnetwork.

[0103] The smallest scale fused feature map is downsampled twice to be named P6 and P7, and the downsampled results P6 and P7 are concatenated with the radar feature maps generated by the results of the last two branches of the radar feature extraction network, and are output to the target classification and regression subnetwork.

[0104] Step S8: a target classification and regression subnetwork is constructed to obtain a target detection result, and the size, position and classification result of the detected target are output.

[0105] Specifically, the five scale fused feature maps of step S7 are subjected to a convolution with a size of the number of prior boxes*the number of classification categories through the target classification subnetwork, and the prediction of each prior box for different categories is output.

[0106] The individual scale fusion feature map of step S7 is outputted through a target regression sub-network and a convolution of a size of prior frame number*4 once, and the adjustment of each prior frame is outputted.

[0107] The target classification sub-network is 4 times of 256 channel convolution, and the target regression sub-network is 4 times of 256 channel convolution.

[0108] In summary, the present application has the following advantages and beneficial effects:

[0109] 1. The present application aims at the problem of inconsistency between millimeter wave radar and image two modes, adopts a similarity fusion method, unifies the two modes, introduces the feature layer containing radar and image semantic information into the target detection network through the similarity fusion module, effectively utilizes the detection ability of millimeter wave radar to small targets in bad weather, and improves the detection effect.

[0110] 2. The method of millimeter wave radar point cloud plane expansion is used, the sparse millimeter wave radar point cloud is expanded horizontally and vertically at the same time, the sparsity of the radar point cloud is improved, and the semantic information contained in the feature layer is increased.

[0111] 3. A lightweight millimeter wave radar feature extraction network is designed, and the problem that the traditional serial network continuously reduces the semantic information contained in the radar point cloud is improved.

[0112] The above description is a detailed description of the preferred embodiment of the present application, but the embodiment is not used to limit the scope of the patent application of the present application, and any equivalent changes or modifications made under the technical spirit disclosed by the present application should belong to the patent scope covered by the present application.

Claims

1. A target detection method based on the fusion of millimeter-wave radar and visual features, characterized in that, Includes the following steps: Acquire calibrated millimeter-wave radar point cloud and camera image, and project the millimeter-wave radar point cloud onto the image pixel plane through three-dimensional spatial coordinate transformation; Vertical and horizontal planar expansion processing is performed on the millimeter-wave radar point cloud projected onto the image pixel plane. A radar feature extraction network is constructed to extract features from millimeter-wave radar point clouds, resulting in radar feature maps. An image feature extraction network is constructed to extract features from camera images, resulting in image feature maps. A similarity calculation module is constructed to calculate the similarity between radar feature maps and image feature maps, thereby obtaining radar feature similarity maps and image feature similarity maps. A similarity fusion module is constructed to sum the radar feature similarity map and the image similarity map pixel by pixel to obtain a fused similarity map; A feature pyramid network is constructed, and the fused similarity map is concatenated with the image feature map to obtain a fused feature map. The fused feature map is then sampled and concatenated with the radar feature map. Construct a target classification and regression subnetwork to obtain target detection results, and output the size, location, and classification results of the detected targets; A similarity calculation module is constructed to calculate the similarity between the radar feature map and the image feature map, resulting in a radar feature similarity map and an image feature similarity map. Specifically: For each channel in the radar feature map at each scale, perform an adaptive max pooling operation to compress the radar feature map into radar feature blocks; For each channel in the feature map of each scale, perform adaptive average pooling to compress the image feature map into image feature blocks; The similarity between radar feature maps and radar feature blocks, and between image feature maps and image feature blocks are calculated to obtain point-to-block similarity maps of different feature maps. The formula for calculating similarity is: In the formula, x i For any point in the radar feature map or image feature map, x j For any point in a radar feature block or image feature block, f(x) i f(x) represents the size of any point in the radar feature map or image feature map. j ) represents the size of any point in a radar feature block or image feature block, and Sim represents the similarity.

2. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, The millimeter-wave radar point cloud is projected onto the image pixel plane through a three-dimensional spatial coordinate transformation, specifically as follows: Transform the millimeter-wave radar point cloud from the radar coordinate system to the world coordinate system centered on the camera; The conversion model is as follows: In the formula, X W Y W Z W For world coordinates, X R Y R Z R H represents the radar coordinate system coordinates. RW For radar coordinate system coordinates Y R Relative world coordinate system coordinates Y W The distance Z RW For radar coordinate system coordinates Z R Relative world coordinate system coordinates Z W The distance; The coordinates of the millimeter-wave radar point cloud in the world coordinate system are transformed into the pixel coordinate system via the camera coordinate system and the image coordinate system; The conversion model is as follows: In the formula, M1 and M2 are the camera intrinsic and extrinsic parameter matrices, u and v are pixel coordinates, and Z is the pixel coordinate system coordinate. C These are the coordinates in the camera coordinate system.

3. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, The millimeter-wave radar point cloud projected onto the image pixel plane undergoes vertical and horizontal planar expansion processing, specifically as follows: Vertical strut expansion is performed on the projected radar points. Distance, reflectivity, and radial relative velocity are selected as three-channel information to generate a three-channel blank feature map with the same size as the camera image. The distance, reflectivity, and radial relative velocity contained in the projected radar points are filled with the corresponding point size of the blank feature map. Then, radar points at a given height above the point are projected onto the blank feature map plane. The vertical expansion line of the point cloud is obtained by connecting and filling the pixels between the two points. The horizontal uncertainty expansion is performed on the projected radar points. For the blank feature map after vertical expansion, it is assumed that the azimuth measurement of each radar point is Gaussian distributed. The measured azimuth is used as the average value, and the accuracy in the radar technical manual is used as the standard deviation. The vertical expansion line of each point is horizontally expanded according to the Gaussian distribution. The maximum value is used when the radar point expansions overlap.

4. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, A radar feature extraction network is constructed to extract features from millimeter-wave radar point clouds, resulting in a radar feature map. Specifically: For each radar point, range, reflectivity, and radial relative velocity are selected as three-channel radar characteristics; The three-channel radar feature map is adjusted by convolution operation with a 1*1 kernel. The three-channel radar feature map is downsampled to the same size as the corresponding scale image feature map through an equivalent convolution operation.

5. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 4, characterized in that, The radar feature extraction network adopts a parallel seven-branch structure. The first five branches are used to downsample the radar feature map to the same size as the corresponding scale image feature map. The last two branches downsample the radar feature map for use in the feature pyramid network. Each branch includes a 1*1 convolutional layer and an equivalent convolution that adjusts the radar feature map to the same size as the corresponding scale image feature map.

6. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, Construct an image feature extraction network to extract features from camera images and obtain image feature maps, specifically: The VGG16 convolutional neural network is used as the image feature extraction network. The VGG16 consists of 5 blocks, with 13 convolutional layers and 3 fully connected layers. The camera RGB image is input into the VGG16, and the VGG16 outputs image feature maps at different scales.

7. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, A similarity fusion module is constructed to sum the radar feature similarity map and the image similarity map pixel by pixel to obtain a fused similarity map, specifically as follows: The feature similarity maps of different channels of radar at each scale are concatenated to generate radar feature similarity maps. The radar feature similarity map and the image feature similarity map are summed pixel by pixel, and then the number of channels is adjusted through a 1*1 convolutional layer to generate a fused similarity map. The fused similarity map is concatenated with image feature maps of different scales generated in VGG16 to generate fused feature maps of different scales.

8. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, A feature pyramid network is constructed, and the fused similarity map is concatenated with the image feature map to obtain a fused feature map. This fused feature map is then sampled and concatenated with the radar feature map, specifically as follows: Starting from the smallest scale, the feature maps fused from different scales are upsampled twice through a feature pyramid network, and the upsampling method is transposed convolution; The minimum scale fused feature map and the fused feature map after each step of upsampling are concatenated with the radar feature map of the same scale and output to the target classification and regression subnetwork. The minimum scale fused feature map is downsampled twice, and the fused feature map after the two downsamplings is concatenated with the radar feature map of the same scale and output to the target classification and regression sub-network.

9. The target detection method based on the fusion of millimeter-wave radar and visual features according to claim 1, characterized in that, Construct a target classification and regression subnetwork to obtain target detection results, and output the size, location, and classification result of the detected target, specifically as follows: The feature maps fused at various scales are passed through a target classification sub-network and a convolution with a size equal to the number of prior boxes multiplied by the number of classification categories, to output the prediction of each prior box for different categories. The feature maps fused at various scales are passed through the target regression sub-network and a convolution of size 4 times the number of prior boxes to output the adjustment of each prior box. The target classification subnetwork consists of four 256-channel convolutions, and the target regression subnetwork consists of four 256-channel convolutions.

Images