Traffic target identification method based on 4D millimeter wave radar and visual fusion

Abstract

The invention relates to the field of computer vision, in particular to a traffic target recognition method based on 4D millimeter wave radar and vision fusion, which comprises the following steps: clustering radar point cloud data by adopting a density-based clustering algorithm, and expanding the center coordinate of each cluster output by the clustering algorithm to obtain an expansion frame; converting the extension value into a two-dimensional visual coordinate system based on a conversion matrix between a radar coordinate and a camera coordinate to obtain an interested target frame under the visual coordinate system; fusing the interested target frame in the visual coordinate system with image data collected by a camera, and taking a fused feature map as the input of a YOLOV8 target detection model; the YOLOV8 target detection model establishes detection branches according to the number of interested target frames, and each branch detects an image of one interested frame part in the fused feature map; according to the method, all-weather sensing work can be realized, the traffic target identification speed is high, and the precision is high.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a traffic target recognition method based on 4D millimeter wave radar and vision fusion. Background Art

[0002] Traffic target detection is one of the important technologies for implementing intelligent transportation systems. With the continuous improvement of intelligent driving levels, new challenges have been raised for roadside traffic target detection technology. Traditional traffic target detection systems use pure visual solutions, which are not able to distinguish longitudinal traffic targets on the road and are affected by uncontrollable environmental factors. In order to solve this problem, many scholars have used various solutions, such as camera and lidar fusion, camera and radar fusion, etc. However, due to issues such as sensor characteristics and cost, most solutions cannot meet the actual production requirements.

[0003] The roadside sensors used in existing vehicle-road cooperative technologies include: laser radar, millimeter-wave radar, and camera sensors. Laser radar is a new type of radar sensor. Its transmitting and receiving signals are lasers with short wavelengths. The radar has a long detection distance and high accuracy, and can image the specific contours of the detected objects, but it is expensive. In addition, due to the weak diffraction ability of laser signals, the detection accuracy of laser radar will be greatly affected in the presence of bad weather, strong light interference, or obstacles. Camera sensors are relatively low in cost, and the information processing technology based on camera-generated images is relatively mature, but the sensor has the disadvantages of large data volume and serious environmental interference. Millimeter-wave radar is a radar sensor that works in the millimeter-wave frequency band. Traditional millimeter-wave radar sensors are relatively low in cost and can detect the speed, distance, and direction angle of the target. Since the electromagnetic waves in the millimeter-wave band have strong penetration capabilities for rain, fog, dust, etc., the radar has the ability to work all day long, but its detection accuracy is lower than that of laser radar, and it cannot image the contours of the target and the static environment. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention proposes a traffic target recognition method based on 4D millimeter wave radar and vision fusion, which specifically includes the following steps:

[0005] The point cloud data set collected by the 4D millimeter-wave radar installed above the road is clustered using a density-based clustering algorithm, and the center coordinates of each cluster output by the clustering algorithm are expanded to obtain an extended frame;

[0006] Based on the transformation matrix between radar coordinates and camera coordinates, the extended value is transformed into a 2D visual coordinate system to obtain the target box of interest in the visual coordinate system;

[0007] The object of interest frame in the visual coordinate system is fused with the image data collected by the camera, and the fused feature map is used as the input of the YOLOV8 object detection model;

[0008] The YOLOV8 target detection model establishes detection branches according to the number of target boxes of interest, and each branch detects the image of a box of interest in the fused feature map;

[0009] The detection results of each branch and the fused feature map are recombined to obtain the final detection results output by the YOLOV8 target detection model.

[0010] Preferably, when obtaining the conversion matrix between the radar coordinates and the camera coordinates, the conversion matrix between the radar coordinates and the camera coordinates is obtained by using the collected N groups of data points, that is, the collected N groups of data points are substituted into the conversion equation between the radar coordinates and the image coordinates to obtain the conversion matrix, and the i-th radar point P i =[x i ,y i ,z i ] T Mapped to point Q in camera coordinates i =[u i ,v i ] T The process includes:

[0011]

[0012] Among them, h 11 is the element in the first row and first column of the transformation matrix; h 12 is the element in the first row and second column of the transformation matrix; h 13 is the element in the first row and third column of the transformation matrix; h 14 is the element in the first row and fourth column of the transformation matrix; h 21 is the element in the second row and first column of the transformation matrix; h 22 is the element in the second row and second column of the transformation matrix; h 23 is the element in the second row and third column of the transformation matrix; h 24 is the element in the second row and fourth column of the transformation matrix; h 31 is the element in the third row and first column of the transformation matrix; h 32 is the element in the third row and second column of the transformation matrix; h 33 is the element in the third row and third column of the transformation matrix; h 34 is the element in the third row and fourth column of the transformation matrix.

[0013] Preferably, the process of filtering and denoising the point cloud data collected by the 4D millimeter wave radar before clustering includes:

[0014] Filter the collected radar point data and only retain the point cloud data within the road area;

[0015] Use conditional filters to remove noise data within a set range.

[0016] Preferably, data enhancement is performed on the point cloud data set before clustering, that is, if the point cloud coordinates of the current frame are (X, Y, Z), when the sampling time interval is 0.04s, and the speed of the current point is V, the point cloud coordinates of the next frame predicted based on the current point cloud are expressed as:

[0017]

[0018] Among them, (X1, Y1, Z1) is the next frame coordinate predicted based on the current frame coordinate; V X 、V Y 、V Z They respectively represent the components of the velocity V of the current point on the X-axis, Y-axis, and Z-axis of the coordinate system.

[0019] Preferably, for the lth target cluster center coordinate value (x l ,y l ,z l ) is expanded, that is, the coordinate value of the target cluster center is expanded up, down, left, and right to obtain four expansion points, and the expansion points are connected to obtain a rectangular expansion box, which is expressed as:

[0020]

[0021] in, The coordinates of the jth extension point representing the coordinate value of the lth target cluster center, j∈{1,2,3,4}; (X, Y, 0) is the preset size of the extension box, and C is the extension constant.

[0022] Preferably, the process of fusing the target frame of interest in the visual coordinate system with the image data collected by the camera includes:

[0023] The cascaded convolutional layers, convolutional blocks and convolutional units are used to process the target frame of interest in the visual coordinate system and the image data collected by the camera in sequence to obtain a first feature map and a second feature map;

[0024] The first feature map and the second feature map are fused by multiplication to obtain a third feature map;

[0025] The third feature map is normalized as data obtained by fusing the target frame of interest in the visual coordinate system with the image data collected by the camera;

[0026] Among them, the convolution layer is composed of cascaded convolution units, normalization layers and activation function layers, and the convolution block is composed of multiple cascaded convolution layers.

[0027] Preferably, the processing of data by each branch of the YOLOV8 target detection model includes the following steps:

[0028] Extract the image of the frame of interest from the fused feature map, reshape the image of the frame of interest to a specified size using a shaping function, and obtain the fourth feature map corresponding to each frame of interest;

[0029] The fourth feature map is processed by the cascaded first convolution layer, the second convolution layer, the first C2f module, the third convolution layer, the second C2f module, the fourth convolution layer, the SPPF module, the CSP module and the convolution layer to obtain a fifth feature map;

[0030] The fifth feature map is processed by the Detect module and then restored to the size of the image of the frame of interest using a shaping function;

[0031] The detection results of each interesting frame part and the fused feature map are recombined to obtain the detection result;

[0032] Among them, the Detect module includes two cascaded convolutional layers and a convolutional unit.

[0033] Preferably, the processing of the feature map by the C2f module includes:

[0034] The input feature map is processed by using the cascaded convolutional layers, convolutional blocks, and convolutional units to obtain a sixth feature map;

[0035] The sixth feature map is fused with the feature map processed by a convolution unit through multiplication;

[0036] The fused feature map is processed by the normalization layer, activation function layer and convolution layer in sequence as the output of the C2f module.

[0037] Preferably, the SPPF module processes the feature map including:

[0038] After processing the input feature map using a convolutional layer, it is then processed using three maximum pooling layers;

[0039] The feature map obtained by the pooling layer and the feature map output by the convolution layer are spliced ​​together and processed by a convolution layer to obtain the feature map output by the SPPF module.

[0040] Preferably, the processing of the feature map by the CSP module includes:

[0041] The input feature map is processed by using the cascaded convolutional layer, residual block and convolutional unit to obtain the seventh feature map, the seventh feature map is fused with the input feature map by multiplication, and the feature map output by the CSP module is obtained by processing the cascaded normalization layer, activation function layer and convolutional layer;

[0042] Among them, the residual block is composed of multiple cascaded residual layers, and each residual layer processes the input feature map through two cascaded convolutional layers, and then fuses the input feature map through multiplication as the output of the residual layer. The present invention is applicable to the field of vehicle-road collaboration. By using multiple sensors to collect road traffic environment data in real time, and then using a targeted pre-trained visual detection end model, and then improving the model to speed up the calculation speed, the advantages of multi-sensors are maximized to meet the requirements of roadside equipment in the vehicle-road coordination system around the clock. By using a multi-transmitter and multi-receiver 4D millimeter-wave radar and camera to identify road traffic targets, and then using a multi-frame fusion strategy to improve the sparseness of the millimeter-wave radar point cloud, and then expanding and converting the central value of the cluster to obtain a visual area of ​​interest with high accuracy, and then decomposing the detection task and reducing the computing power of the image tensor to improve the detection speed. The method of the present invention can sense all-weather, is not affected by external environmental factors, has the characteristics of fast and high accuracy in identifying traffic targets, and can be well deployed on roadside traffic equipment, which is very suitable for the field of vehicle-road collaboration and provides a certain methodological basis for the subsequent development of vehicle-road collaboration. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a traffic target recognition method based on 4D millimeter wave radar and vision fusion of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall structure of the YOLOV8 target detection model of the present invention;

[0045] Figure 3 It is a detailed structural diagram of the YOLOV8 target detection model of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention proposes a traffic target recognition method based on 4D millimeter wave radar and vision fusion, which specifically includes the following steps:

[0048] The point cloud data set collected by the 4D millimeter-wave radar installed above the road is clustered using a density-based clustering algorithm, and the center coordinates of each cluster output by the clustering algorithm are expanded to obtain an extended frame;

[0049] Based on the transformation matrix between radar coordinates and camera coordinates, the extended value is transformed into a 2D visual coordinate system to obtain the target box of interest in the visual coordinate system;

[0050] The object of interest frame in the visual coordinate system is fused with the image data collected by the camera, and the fused feature map is used as the input of the YOLOV8 object detection model;

[0051] The YOLOV8 target detection model establishes detection branches according to the number of target boxes of interest, and each branch detects the image of a box of interest in the fused feature map;

[0052] The detection results of each branch and the fused feature map are recombined to obtain the final detection results output by the YOLOV8 target detection model.

[0053] The overall process implemented by the present invention is as follows Figure 1 As shown, a 4D millimeter-wave radar and a camera are installed on a gantry above the road, and the 4D millimeter-wave radar is used to collect traffic target point cloud data; the sparseness of the 4D millimeter-wave point cloud is compensated by fusing the collected point cloud data for multiple frames, that is, the collected point cloud data is enhanced; the fused point cloud is clustered with an adaptive threshold to obtain the target value, which is the cluster center coordinate of each cluster. After the target expansion conversion, the obtained clustered target is converted to the visual end for detection and recognition; finally, the visual target detected and recognized is fused and matched with the radar cluster target to obtain the output result. The specific implementation process is as follows:

[0054] 1. Data acquisition and preprocessing

[0055] The 4D millimeter wave radar and camera installed above the road are used to collect the surrounding environment in real time, and then transmitted back to the vehicle-mounted MEC system through Socket communication. The point cloud data is parsed, and the point cloud data transmitted back to MEC is parsed according to its data format to obtain the real-time road conditions and various traffic target information scanned; the image data is parsed by setting the road traffic camera and connecting MEC directly through a network cable or using RTSP data stream to push and pull the stream.

[0056] After the radar and camera equipment are fixedly installed in this embodiment, it is necessary to calibrate the built-in parameters of the radar and camera and the external conversion matrix to complete the subsequent data fusion, that is, to obtain the conversion matrix between the image data and the point cloud data according to the acquired image and point cloud data points. Those skilled in the art can use any method to convert the point cloud data coordinates into the image data coordinates. In this embodiment, only one implementation method is provided as a reference. The acquisition process of the conversion matrix specifically includes the following steps:

[0057] Make a black and white camera calibration board to calibrate the built-in parameters of the camera, thereby reducing image distortion caused by the camera;

[0058] By taking sampling values ​​in the radar and the camera at the same time, and sampling by constantly changing the position of the sampling experiment object, multiple pairs of corresponding radar point data and camera target point data can be obtained;

[0059] By taking N data pairs, the conversion matrix between radar coordinates and camera coordinates is obtained through the collected N groups of data points. That is, the collected N groups of data points are substituted into the conversion equation between radar coordinates and image coordinates to obtain the conversion matrix. i =[x i ,y i ,z i ] T Mapped to point Q in camera coordinates i =[u i ,v i ] T The process includes:

[0060]

[0061] in, represents the transformation matrix between radar coordinates and image coordinates, h 11 is the element in the first row and first column of the transformation matrix; h 12 is the element in the first row and second column of the transformation matrix; h 13 is the element in the first row and third column of the transformation matrix; h 14 is the element in the first row and fourth column of the transformation matrix; h 21 is the element in the second row and first column of the transformation matrix; h 22 is the element in the second row and second column of the transformation matrix; h 23 is the element in the second row and third column of the transformation matrix; h 24 is the element in the second row and fourth column of the transformation matrix; h 31 is the element in the third row and first column of the transformation matrix; h 32 is the element in the third row and second column of the transformation matrix; h33 is the element in the third row and third column of the transformation matrix; h 34 is the element in the third row and fourth column of the transformation matrix.

[0062] In order to eliminate the noise of the original point cloud data, the collected original point cloud data is filtered and the sparseness is compensated by fusing multiple frames of data. The specific steps include:

[0063] The original point cloud data is filtered through a straight-through filter, and the filtering range is the road space area. Through the straight-through filtering, the effective point cloud data can be extracted, and the noise data that does not meet the normal range is removed according to the data after the straight-through filtering.

[0064] The point cloud data after filtering is the data in the actual required range, such as Figure 2 The multi-frame data fusion processing flow: the data of the frame is added to the speed compensation calculation to obtain the predicted next frame data, and the predicted frame data is filled into the next frame data to make up for the sparseness; and the next frame data is filled into the next frame again, so that each frame except the first frame can perform multi-frame clustering analysis, and the original point cloud set is enhanced according to the predicted point cloud data, that is, the number of point clouds in the original point cloud set is increased.

[0065] The processed radar point cloud data is clustered by using a density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) with an adaptive threshold to obtain the target value, and then the conversion matrix is ​​used to obtain the accurate extended target. The adaptive threshold DBSCAN clustering selects a clustering method by comparing different types of clustering methods, and selects the clustering effect with the adaptive threshold distance based on the characteristics of different sampling data of the same target at different distances. In the present invention, all point cloud data are used as object data input in the DBSCAN algorithm, and the point cloud is clustered to output the generated clusters. The present invention does not limit the setting of parameters in the DBSCAN algorithm, and those skilled in the art can set them according to experience.

[0066] If the coordinate value of the lth target cluster center obtained by the DBSCAN algorithm is (x l ,y l ,z l ), an expansion frame is obtained by expanding the coordinates. In this embodiment, the Z axis of the expansion frame is fixed (those skilled in the art can choose to fix the X axis or Y axis according to the actual scenario), that is, the coordinate values ​​of the target cluster center are expanded up, down, left, and right to obtain four expansion points, and the expansion points are connected to obtain a rectangular expansion frame. The expansion frame is represented as:

[0067]

[0068] in, The coordinates of the jth extension point representing the coordinate value of the lth target cluster center, j∈{1,2,3,4}; (X, Y, 0) is the preset size of the extension box, and C is the extension constant.

[0069] After obtaining the expansion frame, the image content corresponding to the position of the expansion frame is used as the region of interest, and the target detection is performed through the YOLOV8 target detection model. In the present invention, the point cloud data is first fused with the image data, and then the corresponding regions of interest are extracted from the fused image data according to the expansion frame, and each region of interest is respectively subjected to feature extraction and target recognition through the score of a YOLOV8 target detection model. Finally, the fused image and the recognition results of each region of interest are recombined to obtain the final recognition result. Next, this embodiment sequentially describes the image processing process in the YOLOV8 target detection model of the present invention.

[0070] (I) Fusion of the target frame of interest in the visual coordinate system and the image data collected by the camera

[0071] In this embodiment, if Figure 3 ,The process of fusing the target frame of interest in the visual coordinate system with the image data collected by the camera includes:

[0072] A cascaded convolutional layer (CBL), a convolutional block (composed of 2x CBL cascades, where x is the number of residual units in the residual block) and a convolutional unit (Conv) are used to process the target frame of interest in the visual coordinate system and the image data collected by the camera to obtain a first feature map and a second feature map;

[0073] The first feature map and the second feature map are fused by multiplication to obtain a third feature map;

[0074] The third feature map is normalized as data obtained by fusing the target frame of interest in the visual coordinate system with the image data collected by the camera;

[0075] Among them, the convolution layer is composed of cascaded convolution units, normalization layers and activation function layers, and the convolution block is composed of 2x cascaded convolution layers.

[0076] (II) Branches of the YOLOV8 target detection model

[0077] In this embodiment, the processing of data by each branch of the YOLOV8 target detection model includes the following steps:

[0078] Extract the image of the interesting box from the fused feature map, that is Figure 3In the ROI unit with a size of n×n×3, a reshaping function (Resize) is used to reshape the image of the interest box to a specified size. In this embodiment, the specified size is 320×320×3, and the fourth feature map corresponding to each interest box is obtained respectively;

[0079] The feature extraction network extracts the fifth feature map from the fourth feature map, and the feature extraction network consists of a cascaded first convolutional layer, a second convolutional layer, a first C2f (CSP Bottleneck with 2Convolutions) module, a third convolutional layer, a second C2f module, a fourth convolutional layer, a SFFP (Spatial Pyramid Pooling–Fast) module, a CSP (Cross Stage Partial) module, and a convolutional layer;

[0080] The fifth feature map is processed by the Detect module and then restored to the size of the image of the frame of interest using a shaping function;

[0081] The detection results of each interesting frame part and the fused feature map are recombined to obtain the detection result;

[0082] Among them, the Detect module includes two cascaded convolutional layers and a convolutional unit. The processed feature map is regressed through CIOU+DFL to obtain the detection result; each convolutional layer is composed of a cascaded convolutional unit, a normalization layer (BN layer), and an activation function layer. In this embodiment, the activation function is the Leaky relu function.

[0083] Preferably, the processing of the feature map by the C2f module includes:

[0084] The sixth feature map is obtained by processing the input feature map using cascaded convolutional layers, residual blocks and convolutional units. The residual block is composed of x cascaded residual layers. The input of each residual layer is processed by two cascaded convolutional layers and then fused with the input of the residual layer by multiplication as the output of the residual layer.

[0085] The sixth feature map and the input feature map of the C2f module are fused by multiplying the feature map processed by a convolution unit;

[0086] The fused feature map is processed by the normalization layer, activation function layer and convolution layer in sequence as the output of the C2f module.

[0087] Preferably, the SPPF module processes the feature map including:

[0088] After processing the input feature map using a convolutional layer, it is then processed using three maximum pooling layers;

[0089] The feature map obtained by the pooling layer and the feature map output by the convolution layer are spliced ​​together and processed by a convolution layer to obtain the feature map output by the SPPF module.

[0090] Preferably, the processing of the feature map by the CSP module includes:

[0091] The input feature map is processed by using the cascaded convolutional layer, residual block and convolutional unit to obtain the seventh feature map, the seventh feature map is fused with the input feature map by multiplication, and the feature map output by the CSP module is obtained by processing the cascaded normalization layer, activation function layer and convolutional layer;

[0092] Among them, the residual block is composed of multiple cascaded residual layers. Each residual layer processes the input feature map through two cascaded convolutional layers and then fuses it with the input feature map by multiplication as the output of the residual layer.

[0093] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A traffic target recognition method based on 4D millimeter wave radar and vision fusion, characterized in that: The specific steps include: The point cloud data set collected by the 4D millimeter-wave radar installed above the road is clustered using a density-based clustering algorithm, and the center coordinates of each cluster output by the clustering algorithm are expanded to obtain an extended frame; Based on the transformation matrix between radar coordinates and camera coordinates, the extended value is transformed into a 2D visual coordinate system to obtain the target box of interest in the visual coordinate system; The object of interest frame in the visual coordinate system is fused with the image data collected by the camera, and the fused feature map is used as the input of the YOLOV8 object detection model; The YOLOV8 target detection model establishes detection branches according to the number of target boxes of interest, and each branch detects the image of a box of interest in the fused feature map; The detection results of each branch and the fused feature map are recombined to obtain the final detection results output by the YOLOV8 target detection model.

2. According to claim 1, a traffic target recognition method based on 4D millimeter wave radar and vision fusion is characterized in that: When obtaining the conversion matrix between radar coordinates and camera coordinates, the conversion matrix between radar coordinates and camera coordinates is obtained through the collected N groups of data points, that is, the collected N groups of data points are substituted into the conversion equation between radar coordinates and image coordinates to obtain the conversion matrix, and the i-th radar point P i =[x i ,y i ,z i ] T Mapped to point Q in camera coordinates i =[u i ,v i ] T The process includes: Among them, h 11 is the element in the first row and first column of the transformation matrix; h 12 is the element in the first row and second column of the transformation matrix; h 13 is the element in the first row and third column of the transformation matrix; h 14 is the element in the first row and fourth column of the transformation matrix; h 21 is the element in the second row and first column of the transformation matrix; h 22 is the element in the second row and second column of the transformation matrix; h 23 is the element in the second row and third column of the transformation matrix; h 24 is the element in the second row and fourth column of the transformation matrix; h 31 is the element in the third row and first column of the transformation matrix; h 32 is the element in the third row and second column of the transformation matrix; h 33 is the element in the third row and third column of the transformation matrix; h 34 is the element in the third row and fourth column of the transformation matrix.

3. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 1 is characterized in that: The process of filtering and denoising the point cloud data collected by the 4D millimeter wave radar before clustering includes: Filter the collected radar point data and only retain the point cloud data within the road area; Use conditional filters to remove noise data within a set range.

4. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 1 or 3, characterized in that: Before clustering, the point cloud dataset is enhanced. That is, if the point cloud coordinates of the current frame are (X, Y, Z), when the sampling time interval is 0.04s, and the speed of the current point is V, the point cloud coordinates of the next frame predicted based on the current point cloud are expressed as: Among them, (X1, Y1, Z1) is the next frame coordinate predicted based on the current frame coordinate; V X 、V Y 、V Z They respectively represent the components of the velocity V of the current point on the X-axis, Y-axis, and Z-axis of the coordinate system.

5. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 1 is characterized in that: For the lth target cluster center coordinate value (x l ,y l ,z l ) is expanded, that is, the coordinate value of the target cluster center is expanded up, down, left, and right to obtain four expansion points, and the expansion points are connected to obtain a rectangular expansion box, which is expressed as: in, The coordinates of the jth extension point representing the coordinate value of the lth target cluster center, j∈{1,2,3,4}; (X, Y, 0) is the preset size of the extension box, and C is the extension constant.

6. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 1 is characterized in that: The process of fusing the target frame of interest in the visual coordinate system with the image data collected by the camera includes: The cascaded convolutional layers, convolutional blocks and convolutional units are used to process the target frame of interest in the visual coordinate system and the image data collected by the camera in sequence to obtain a first feature map and a second feature map; The first feature map and the second feature map are fused by multiplication to obtain a third feature map; The third feature map is normalized as data obtained by fusing the target frame of interest in the visual coordinate system with the image data collected by the camera; Among them, the convolution layer is composed of cascaded convolution units, normalization layers and activation function layers, and the convolution block is composed of multiple cascaded convolution layers.

7. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 1 is characterized in that: The processing of data by each branch of the YOLOV8 target detection model includes the following steps: Extract the image of the frame of interest from the fused feature map, reshape the image of the frame of interest to a specified size using a shaping function, and obtain the fourth feature map corresponding to each frame of interest; The fourth feature map is processed by the cascaded first convolution layer, the second convolution layer, the first C2f module, the third convolution layer, the second C2f module, the fourth convolution layer, the SPPF module, the CSP module and the convolution layer to obtain a fifth feature map; The fifth feature map is processed by the Detect module and then restored to the size of the image of the frame of interest using a shaping function; The detection results of each interesting frame part and the fused feature map are recombined to obtain the detection result; Among them, the Detect module includes two cascaded convolutional layers and a convolutional unit.

8. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 7 is characterized in that: The C2f module processes the feature map as follows: The input feature map is processed by using the cascaded convolutional layers, convolutional blocks, and convolutional units to obtain a sixth feature map; The sixth feature map is fused with the feature map processed by a convolution unit through multiplication; The fused feature map is processed by the normalization layer, activation function layer and convolution layer in sequence as the output of the C2f module.

9. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 7 is characterized in that: The SPPF module processes the feature map as follows: After processing the input feature map using a convolutional layer, it is then processed using three maximum pooling layers; The feature map obtained by the pooling layer and the feature map output by the convolution layer are spliced ​​together and processed by a convolution layer to obtain the feature map output by the SPPF module.

10. The method for traffic target recognition based on 4D millimeter wave radar and vision fusion according to claim 7, characterized in that: The CSP module processes the feature map as follows: The input feature map is processed by using the cascaded convolutional layer, residual block and convolutional unit to obtain the seventh feature map, the seventh feature map is fused with the input feature map by multiplication, and the feature map output by the CSP module is obtained by processing the cascaded normalization layer, activation function layer and convolutional layer; Among them, the residual block is composed of multiple cascaded residual layers. Each residual layer processes the input feature map through two cascaded convolutional layers and then fuses it with the input feature map by multiplication as the output of the residual layer.

Images