Vehicle-end target classification detection method and vehicle-end radar and visual fusion device

Abstract

The application relates to the technical field of automatic driving, and particularly discloses a vehicle-end target classification detection method based on millimeter wave radar and vision fusion and a vehicle-end radar-vision fusion device, which comprises the following steps: acquiring a vehicle-end vision image and vehicle-end millimeter wave radar point cloud data respectively; inputting the vehicle-end vision image and the vehicle-end millimeter wave radar point cloud data into an image classification and radar attention module respectively to obtain a preliminary classification feature map, wherein the image classification and radar attention module can process the vehicle-end millimeter wave radar point cloud data and then use the processed data as a gating module to perform feature fusion with the vehicle-end vision image; performing small target detection processing on the preliminary classification feature map to obtain a refined classification feature map; and inputting the refined classification feature map into a classification detection network to obtain a vehicle-end classification category and a classification target frame. The vehicle-end target classification detection method based on millimeter wave radar and vision fusion provided by the application improves the vehicle-end target detection precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a vehicle-end target classification detection method based on millimeter wave radar and vision fusion and a vehicle-end radar-vision fusion device. BACKGROUND

[0002] In recent years, automatic driving has become the frontier and hotspot of the development of the automobile industry. Through technical means such as sensor fusion, the road condition information in the driving process is comprehensively perceived to realize traffic safety and efficiency. Target detection and classification is one of the cores of accurate perception, and is also an important research direction and a highly challenging research topic.

[0003] Currently, the problems to be solved in vehicle-end target detection and classification are: all-weather perception of the front road traffic targets in the driving process, including the position, category, and distance information of vehicles and pedestrians, which has a very high precision requirement. The traditional vehicle-end target detection and classification mostly uses a single sensor technology route for detection, which does not meet the all-weather and high-precision requirements in the industry standard.

[0004] For example, the Chinese patent with publication number CN104573646B discloses a traffic target detection and ranging method based on deep learning. The traffic image data is first normalized and then input into a deep learning network for training to obtain a model, and finally the category, coordinates, distance, and confidence of the detected target are output. However, the applicant finds that the above target detection method only uses single image information, and its stability is easily affected by light conditions, rain, snow, and other environmental disturbances.

[0005] For another example, the Chinese patent application with publication number CN111898439A discloses a traffic scene joint target detection and semantic segmentation method based on deep learning. It still uses a single image sensor, but the overall structure has high computational complexity and poor real-time performance.

[0006] Therefore, how to improve the detection accuracy of vehicle-end targets has become a technical problem to be solved by those skilled in the art. SUMMARY

[0007] The present application provides a vehicle-end target classification detection method based on millimeter wave radar and vision fusion and a vehicle-end radar-vision fusion device, which solves the problem of low vehicle-end target detection accuracy in the related art.

[0008] As a first aspect of the present application, a vehicle-end target classification detection method based on millimeter wave radar and vision fusion is provided, which comprises:

[0009] acquiring vehicle-end vision images and vehicle-end millimeter wave radar point cloud data respectively;

[0010] The vehicle end visual image and the vehicle end millimeter wave radar point cloud data are respectively input into an image classification and radar attention module to obtain a preliminary classification feature map, wherein the image classification and radar attention module can process the vehicle end millimeter wave radar point cloud data as a gating module and perform feature fusion with the vehicle end visual image.

[0011] The preliminary classification feature map is subjected to small target detection processing to obtain a refined classification feature map.

[0012] The refined classification feature map is input into a classification detection network to obtain a vehicle end classification category and a classification target frame.

[0013] Further, the vehicle end visual image and the vehicle end millimeter wave radar point cloud data are respectively input into an image radar fusion module to obtain a preliminary classification feature map, including:

[0014] The vehicle end millimeter wave radar point cloud data is subjected to compensation processing to obtain a radar image.

[0015] The radar image and the vehicle end visual image are subjected to first fusion processing through a radar attention module to obtain a first fusion feature map.

[0016] The first fusion feature map and the vehicle end visual image are subjected to second fusion processing through a channel attention module to obtain a preliminary classification feature map.

[0017] Further, the vehicle end millimeter wave radar point cloud data is subjected to compensation processing to obtain a radar image, including:

[0018] Each frame of vehicle end millimeter wave radar point cloud data is projected into a front view.

[0019] A line in the height direction is generated in the front view, and the pixel values not containing the line are set to 0 as a whole to obtain a radar image, wherein the width of the line in the height direction in the front view occupies 1 pixel to cover all targets.

[0020] Further, the radar image and the vehicle end visual image are subjected to first fusion processing through a radar attention module to obtain a first fusion feature map, including:

[0021] The radar image is subjected to an activation function to obtain a weight matrix with a value range of [0, 1];

[0022] The weight matrix and the vehicle end visual image are subjected to pixel-by-pixel multiplication to obtain a first fusion feature map.

[0023] Further, the first fusion feature map and the vehicle end visual image are subjected to second fusion processing through a channel attention module to obtain a preliminary classification feature map, including:

[0024] After pixel-by-pixel addition fusion processing of the first fusion feature map and the vehicle end visual image, a preliminary classification feature map is obtained.

[0025] Further, the preliminary classification feature map is subjected to small target detection processing to obtain a refined classification feature map, including:

[0026] a feature pyramid module is constructed;

[0027] The preliminary classification feature map is input into the feature pyramid module for small target detection processing to obtain a refined classification feature map.

[0028] Further, the refined classification feature map is input into a classification detection network to obtain a vehicle end classification category and a classification target frame, including:

[0029] a classification detection network is constructed;

[0030] The refined classification feature map is input into the classification detection network;

[0031] A detection result output is obtained according to a detection loss function of the classification detection network, and a classification result output is obtained according to a classification loss function of the classification detection network.

[0032] Further, the classification detection network is constructed, including:

[0033] a cross-modal dataset is prepared, wherein the cross-modal dataset includes a millimeter wave radar point cloud dataset and a visual image dataset, and the cross-modal dataset is divided into a training set and a test set according to a preset ratio;

[0034] The cross-modal dataset is subjected to back propagation training and testing according to an Adam optimizer to obtain a classification detection network.

[0035] Further, the vehicle end target classification detection method based on millimeter wave radar and visual fusion further includes, before the steps of respectively acquiring a vehicle end visual image and a vehicle end millimeter wave radar point cloud dataset:

[0036] The millimeter wave radar information acquisition device and the visual image acquisition device are registered.

[0037] As another aspect of the present application, a vehicle end radar-visual fusion device is provided, which includes a memory and a processor, the memory and the processor are communicatively connected, the memory is used to store computer instructions, and the processor is used to load and execute the computer instructions to realize the vehicle end target classification detection method based on millimeter wave radar and visual fusion described above.

[0038] The application provides a vehicle end target classification detection method based on millimeter wave radar and vision fusion. BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and serve to explain the principles of the application, but are not intended to limit the application.

[0040] Figure 1 The flow chart of the vehicle end target classification detection method based on millimeter wave radar and vision fusion provided by the application.

[0041] Figure 2 The working flow chart of the radar attention module provided by the application.

[0042] Figure 3 The working flow chart of the channel attention module provided by the application.

[0043] Figure 4 The schematic diagram of the target converted from the radar coordinate system to the image coordinate system provided by the application.

[0044] Figure 5 The VGG16+radar attention model structure diagram provided by the application.

[0045] Figure 6 The feature pyramid structure diagram provided by the application.

[0046] Figure 7 The RetinaNet classification and detection network structure diagram provided by the application.

[0047] Figure 8 The overall working flow chart of the vehicle end target classification detection method based on millimeter wave radar and vision fusion provided by the application. DETAILED DESCRIPTION

[0048] It should be noted that the embodiments and features in the embodiments of the application can be combined with each other without conflict. The application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] In order to make the technical personnel of the present application better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by the ordinary skilled in the art without creative labor should belong to the scope of protection of the present application.

[0050] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0051] In the present embodiment, a vehicle end target classification detection method based on fusion of millimeter wave radar and vision is provided, Figure 1 is a flow chart of the vehicle end target classification detection method based on fusion of millimeter wave radar and vision provided according to the embodiments of the present application, as Figure 1 shown, comprising:

[0052] S100, acquiring vehicle end vision image and vehicle end millimeter wave radar point cloud data respectively;

[0053] In the embodiments of the present application, the vehicle end millimeter wave radar point cloud data is collected in real time by the millimeter wave radar information collection device, and the vehicle end vision image is collected in real time by the vision image collection device.

[0054] In some embodiments, the millimeter wave radar information collection device can be a millimeter wave radar device, and the vision image collection device can be a camera.

[0055] S200, inputting the vehicle end vision image and the vehicle end millimeter wave radar point cloud data into an image classification and radar attention module respectively to obtain a preliminary classification feature map, wherein the image classification and radar attention module can process the vehicle end millimeter wave radar point cloud data as a gating module and perform feature fusion with the vehicle end vision image;

[0056] In the embodiments of the present application, as Figure 1 and Figure 8As shown, the radar image and the visual image are respectively and simultaneously input into an image classification and radar attention module (which can be VGG16+ radar attention model), the model contains five modules in total, and the output of each module of the two sensor branches is subjected to feature depth interaction through the radar attention module.

[0057] In some embodiments, the specific step of obtaining the preliminary classification feature map can include:

[0058] S210, compensating the vehicle-end millimeter wave radar point cloud data to obtain a radar image;

[0059] Specifically, each frame of vehicle-end millimeter wave radar point cloud data is projected into a front view.

[0060] A line in the height direction is generated in the front view, and the pixel values not containing the line are set to 0 as a whole to obtain a radar image, wherein the width of the line in the height direction in the front view occupies 1 pixel to cover all targets.

[0061] It should be noted that in the embodiments of the present application, the front view can specifically refer to the driving view, i.e., the in-vehicle view.

[0062] It should be understood that since the millimeter wave radar lacks height information, each frame of millimeter wave radar point cloud is first projected into a front view and is lowered by 50 cm as a whole as compensation for the height of the in-vehicle millimeter wave radar device, and then a line of 3 m is generated in the vertical direction above the point cloud, the width of the line in the front view occupies 1 pixel to basically cover all targets, and the pixel values not containing the line are set to 0 as a whole, the processed image is taken as the input image of the radar channel, and the original RGB image is directly taken as the input image of the visual channel.

[0063] S220, first fusion feature map is obtained by performing first fusion processing on the radar image and the vehicle-end visual image through the radar attention module;

[0064] In some embodiments, it can specifically include:

[0065] The radar image is subjected to an activation function to obtain a weight matrix with a value range of [0, 1];

[0066] The weight matrix and the vehicle-end visual image are subjected to a pixel-by-pixel multiplication operation to obtain a first fusion feature map.

[0067] In the embodiments of the present application, as shown, Figure 2 For the radar attention module, the Softmax activation function is operated on each feature map along the channel axis on the feature input of the radar channel pixel by pixel, and the specific formula of the Softmax activation function can be written as:

[0068]

[0069] wherein z i represents a specific pixel value, c represents the total number of pixels in each feature map, and the activation function can generate a weight matrix in the value range [0, 1].

[0070] Since the values after pixel-by-pixel multiplication are all located in the range [0, 1], it is necessary to prevent the values from being multiplied to be smaller and to be unable to realize feature distinction, so that a channel attention module is added to realize pixel-by-pixel addition, so that each value located in the range [0, 1] before the front can realize obvious distinction of features after increasing a value.

[0071] S230, the first fusion feature map and the vehicle end visual image are fused again through the channel attention module to obtain a preliminary classification feature map.

[0072] In the embodiments of the application, the first fusion feature map and the vehicle end visual image can be fused by pixel-by-pixel addition to obtain a preliminary classification feature map.

[0073] It should be noted that, as Figure 3 shown, for the channel attention module, the average pooling and the maximum pooling are global pooling, and after the two pooling processes are completed, they are simultaneously input into two shared multilayer perceptrons, the sharing ratio is set to 16, and the specific structure is as follows:

[0074] (1) first stretch the feature map after the pooling into a one-dimensional sequence in pixel order, and input 2.

[0075] (2) the first layer adopts the Keras self-multilayer perceptron function Dense, the activation function is Relu, the output channel is the number of one-dimensional sequence elements / sharing ratio, and the input is 3.

[0076] (3) the second layer adopts the Keras self-multilayer perceptron function Dense, the activation function is Relu, the output channel is the number of one-dimensional sequence elements, and the output is an interaction feature.

[0077] The feature after the maximum pooling is input into the shared multilayer perceptron to generate an interaction feature 1, and the feature after the average pooling is input into the shared multilayer perceptron to generate an interaction feature 2.

[0078] S300, the preliminary classification feature map is subjected to small target detection processing to obtain a refined classification feature map.

[0079] In some embodiments, the method can specifically include:

[0080] constructing a feature pyramid module;

[0081] The preliminary classification feature map is input into the feature pyramid module for small target detection processing to obtain a refined classification feature map.

[0082] It should be understood that the feature map generated by the last three modules of the fused visual branch VGG16 model is input into the feature pyramid processing module to improve the detection of small targets.

[0083] S400, the refined classification feature map is input into a classification detection network to obtain a vehicle-end classification category and a classification target frame.

[0084] Specifically, a classification detection network is constructed.

[0085] The refined classification feature map is input into the classification detection network.

[0086] A detection result output is obtained according to a detection loss function of the classification detection network, and a classification result output is obtained according to a classification loss function of the classification detection network.

[0087] Further specifically, the construction of the classification detection network comprises:

[0088] A cross-modal data set is prepared, wherein the cross-modal data set comprises a millimeter wave radar point cloud data set and a visual image data set, and the cross-modal data set is divided into a training set and a test set according to a preset ratio.

[0089] The cross-modal data set is trained and tested by back propagation according to an Adam optimizer to obtain a classification detection network.

[0090] It should be understood that the refined classification feature map (i.e., the five feature maps output by the feature pyramid processing module) is input into a RetinaNet classification and detection network to output a target frame and category information.

[0091] In summary, the vehicle-end target classification and detection method based on millimeter wave radar and visual fusion provided by the application improves the robustness and anti-interference of the algorithm to a great extent by fusing the vehicle-end millimeter wave radar point cloud data and the vehicle-end visual image, using the radar image generated by the radar point cloud as a gating module to realize spatial attention function and provide guidance for visual detection and classification. In addition, the advanced channel attention module is added, which is plug-and-play, and is further fused with the aforementioned radar spatial attention module, so that the algorithm accuracy is further improved, and the accuracy is greatly improved compared with the single visual algorithm in rainy and night conditions.

[0092] In the embodiment of the application, in order to realize data acquisition, the millimeter wave radar and vision fusion based vehicle end target classification detection method further comprises the following steps performed before the steps of acquiring the vehicle end vision image and the vehicle end millimeter wave radar point cloud data respectively:

[0093] The millimeter wave radar information acquisition device and the vision image acquisition device are registered.

[0094] The trained model is deployed to a vehicle end radar-vision fusion device, and the radar point cloud and the vision image corresponding to each frame are input into the model to generate real-time detection frames and classification results.

[0095] Specifically, the millimeter wave radar and camera coordinate system registration standard is rotation and translation.

[0096]

[0097] wherein (x c , y c , z c ) is the millimeter wave radar coordinate system, (x r , y r , z r ) is the image coordinate system, R is a 3x3 conversion matrix, and (t1, t2, t3) is a translation vector.

[0098] The radar-vision coordinate system conversion is performed to map the target point position in the radar coordinate system to the image coordinate system; wherein (X c , Y c , Z c ) is the radar coordinate system, (X r , Y r , Z r ) is the image coordinate system, and is shown as follows:

[0099]

[0100] The space conversion matrix T is:

[0101]

[0102]

[0103] As shown in Figure 4 , it is a schematic diagram of the conversion of a target from a radar coordinate system to an image coordinate system. Wherein, the relative positions of the millimeter wave radar and the camera are illustrated, Figure 4 o-x c y c z c indicates the camera coordinate system, o-x r y r z rThe radar coordinate system is represented by o-uv, and the image array coordinate system is represented by o-uv, with the origin being the upper left corner of the image. The camera coordinate system and the radar coordinate system are measured in meters, and the image array coordinate system is measured in pixels. Figure 4 The midpoint p represents the target point, and p' is the image pixel point corresponding to p.

[0104] During the training process, the missed and false detection samples are used as "difficult examples", and the "difficult example" dataset is used for re-training to obtain the final model, and then the model is used for real-time scene traffic target detection.

[0105] The specific construction principle of the VGG16+radar attention model provided by the embodiment of the application is described in detail below.

[0106] As shown in Figure 5 , the radar input and visual input size is set to 360x640, the radar is a 2-channel image, each channel is the RCS and distance line projection after point cloud projection, and the vision is a RGB 3-channel image.

[0107] For all convolutions, the convolution kernel is set to 3x3, the stride is 1, the padding parameter is same, and the activation function is Relu. The convolution formula is as follows:

[0108]

[0109] f[x,y] represents the pixel value within the convolution kernel size, h[x,y] represents the convolution kernel, and the convolution kernel uses the default 2D convolution kernel value of Keras.

[0110] The Relu activation function formula is as follows:

[0111] f(x)=max(0,x).

[0112] For max pooling, the window is set to 2x2, the stride is set to 2, the maximum pixel of each window is taken, and the image size size becomes 0.5 times the size of the previous step after each stage.

[0113] The output channel number of each stage is set to (32, 64, 128, 512, 512).

[0114] The feature pyramid module provided by the application is described in detail below. Figure 6

[0115] Convolution A sets the kernel to 1x1, the step length to 1, and there is no activation function. For feature inputs 2 and 3, the output channel number is consistent with feature inputs 1 and 2, and for input feature 1, the output channel number does not change; Convolution B is set the same as Convolution A except that the kernel is 3x3, and the output channel number is 256; Convolution C is set to a kernel of 3x3, a step length of 2, and no activation function inside, with an output channel of 256.​

[0116] In the upsampling process, the output resolution of the upsampling feature should be consistent with the input of the guide feature.

[0117] As shown in the RetinaNet classification and detection network structure diagram. Figure 7

[0118] For the input feature to be classified, the stage 1 convolution kernel is set to 3x3, the filter number is 256, and the activation function is Relu. The stage 2 convolution is basically the same as the convolution 1, and the filter number is KxA, where K and A are the number of custom categories and anchor boxes.

[0119] For the input feature to be detected, the filter number is 4A, where 4 represents the coordinates of the anchor box points. For each layer of features, the aspect ratio of the anchor box is set to [1:1, 1:2, 2:1].

[0120] It should be particularly noted that in the backpropagation process, the algorithm adopts different loss function strategies for detection and classification.

[0121] For detection, the Focal loss is used, and the formula is:

[0122] FL(P t )=-(1-P t ) γ log(P t ),

[0123] The adjustment factor γ = 0.2, P t is the classification confidence value of the t specific category. According to the NuScenes category sample difference, the following settings are made: car, 1; bus, 30; motorcycle, 39; truck, 6; trailer, 20; bicycle, 42; pedestrian, 2.

[0124] For classification, the L1 smooth loss is used, and the formula is:

[0125]

[0126] Where Y represents the true value and f(x) represents the predicted value.

[0127] ​Therefore, the millimeter wave radar and vision fusion based vehicle end target classification detection method provided by the present application can improve the robustness and anti-interference of the algorithm to a great extent, and effectively improve the detection accuracy of the vehicle end target, and the accuracy is greatly improved compared with the single image method in extreme weather and dark light conditions, and the all-weather demand is met.

[0128] As another embodiment of the present application, a vehicle end radar and vision fusion device is provided, which comprises a memory and a processor, the memory and the processor are communicatively connected, the memory is used to store computer instructions, and the processor is used to load and execute the computer instructions to realize the millimeter wave radar and vision fusion based vehicle end target classification detection method described above.

[0129] The specific working principle of the vehicle end radar and vision fusion device provided by the present application can be referred to the description of the millimeter wave radar and vision fusion based vehicle end target classification detection method described above, and will not be repeated here.

[0130] It can be understood that the above embodiments are only exemplary embodiments adopted for illustrating the principles of the present application, and the present application is not limited thereto. Various modifications and improvements can be made by those skilled in the art without departing from the spirit and essence of the present application, and these modifications and improvements are also regarded as the protection scope of the present application.

Claims

1. A method for vehicle end target classification detection based on fusion of millimeter wave radar and vision, characterized in that, The method comprises the following steps: acquire vehicle-end visual image and vehicle-end millimeter wave radar point cloud data respectively; input the vehicle-end visual image and the vehicle-end millimeter wave radar point cloud data into an image classification and radar attention module respectively to obtain a preliminary classification feature map, wherein the image classification and radar attention module can process the vehicle-end millimeter wave radar point cloud data as a gating module and perform feature fusion with the vehicle-end visual image; perform small target detection processing on the preliminary classification feature map to obtain a refined classification feature map; input the refined classification feature map into a classification detection network to obtain a vehicle-end classification category and a classification target frame; wherein the step of inputting the vehicle-end visual image and the vehicle-end millimeter wave radar point cloud data into the image classification and radar attention module to obtain the preliminary classification feature map comprises the following steps: perform compensation processing on the vehicle-end millimeter wave radar point cloud data to obtain a radar image; perform first fusion processing on the radar image and the vehicle-end visual image through a radar attention module to obtain a first fusion feature map; perform second fusion processing on the first fusion feature map and the vehicle-end visual image through a channel attention module to obtain the preliminary classification feature map; wherein the image classification and radar attention module comprises a combination of VGG16 and a radar attention model, and the construction process of the VGG16 and the radar attention model comprises the following steps: set the radar input and the visual input size to 360x640, the radar to 2-channel images, each channel to the RCS and distance line projection after point cloud projection, and the visual to an RGB 3-channel image; for all convolutions, set the convolution kernel to 3x3, the stride to 1, the padding parameter to same, and the activation function to Relu, and the convolution formula is as follows: ， represents the pixel value within the convolution kernel size, represents the convolution kernel, the convolution kernel adopts the Keras default 2D convolution kernel value; the Relu activation function formula is as follows: ， for max pooling, set the window to 2x2, the stride to 2, and take the maximum pixel of each window, and the image size size becomes 0.5 times the size of the previous step after each stage; set the output channel number of each stage to (32, 64, 128, 512, 512).

2. The method of claim 1, wherein the method further comprises: the step of performing compensation processing on the vehicle-end millimeter wave radar point cloud data to obtain a radar image comprises the following steps: project each frame of vehicle-end millimeter wave radar point cloud data into a front view; generate a height direction line in the front view, and set the pixel values not containing the line to 0 as a whole to obtain a radar image, wherein the width of the height direction line in the front view occupies 1 pixel to cover all targets.

3. The method of claim 1, wherein the method further comprises: the step of performing first fusion processing on the radar image and the vehicle-end visual image through a radar attention module to obtain a first fusion feature map comprises the following steps: obtain a weight matrix with a value range of [0, 1] by performing an activation function on the radar image; perform a pixel-by-pixel multiplication operation on the weight matrix and the vehicle-end visual image to obtain a first fusion feature map.

4. The method of claim 1, wherein the method further comprises: the step of performing second fusion processing on the first fusion feature map and the vehicle-end visual image through a channel attention module to obtain a preliminary classification feature map comprises the following steps: perform pixel-by-pixel addition fusion processing on the first fusion feature map and the vehicle-end visual image to obtain a preliminary classification feature map.

5. The vehicle end object classification detection method based on fusion of millimeter wave radar and vision according to any one of claims 1 to 4, characterized in that, The preliminary classification feature map is subjected to small target detection processing to obtain a refined classification feature map, including: a feature pyramid module is constructed; the preliminary classification feature map is input into the feature pyramid module for small target detection processing to obtain a refined classification feature map.

6. The vehicle end object classification detection method based on fusion of millimeter wave radar and vision according to any one of claims 1 to 4, characterized in that, The refined classification feature map is input into a classification detection network to obtain a vehicle-end classification category and a classification target frame, including: a classification detection network is constructed; the refined classification feature map is input into the classification detection network; a detection result output is obtained according to a detection loss function of the classification detection network, and a classification result output is obtained according to a classification loss function of the classification detection network.

7. The method of claim 6, wherein the method further comprises: The classification detection network is constructed, including: a cross-modal data set is prepared, wherein the cross-modal data set includes a millimeter wave radar point cloud data set and a visual image data set, and the cross-modal data set is divided into a training set and a test set according to a preset ratio; the cross-modal data set is subjected to back propagation training and testing according to an Adam optimizer to obtain a classification detection network.

8. The method of claim 1, wherein the method further comprises: The vehicle-end target classification detection method based on millimeter wave radar and visual fusion further includes, before the steps of acquiring a vehicle-end visual image and a vehicle-end millimeter wave radar point cloud data, respectively: a millimeter wave radar information acquisition device and a visual image acquisition device are registered.

9. A vehicle end radar and visual fusion device, characterized by The vehicle-end target classification detection method based on millimeter wave radar and visual fusion includes a memory and a processor, which are communicatively connected, the memory is used to store computer instructions, and the processor is used to load and execute the computer instructions to implement the vehicle-end target classification detection method based on millimeter wave radar and visual fusion in any one of claims 1 to 8.

Images