A target recognition method, system and electronic device based on radar and visual fusion

Abstract

The application provides a target identification method and system based on radar and video fusion and electronic equipment, and relates to the technical field of target identification. Video data is subjected to frame processing to obtain a video frame sequence, which is input into a video target identification model to obtain a plurality of video identification targets; the video target identification model is obtained by training a YOLO neural network using historical labeled video data of a to-be-tested intersection; a plurality of radar identification targets are determined according to radar point cloud data; a fusion model is used to fuse the plurality of video identification targets and the plurality of radar identification targets to obtain a target identification result of the to-be-tested intersection. The application improves the accuracy of road target identification and the rationality of formulating a road self-adaptive timing scheme by constructing a fusion model and combining radar data and video data for road target identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target recognition, in particular to a target recognition method and system based on radar and video fusion and electronic equipment. BACKGROUND

[0002] With the development of economy, the number of motor vehicles on the road is increasing, the degree of road congestion is getting bigger and bigger, and the queuing time of vehicles is getting longer and longer. Urban traffic management urgently needs an adaptive signal timing adjustment scheme to alleviate the situation of poor traffic flow on the road. Adaptive timing depends on the road information perception capability. Accurate and timely acquisition of current intersection targets and traffic flow statistics (such as average speed, target vehicle position, and queue length) provides a data basis and basis for timing algorithms.

[0003] The existing road perception scheme mainly includes ground inductive coils, millimeter wave radars, and video analysis. Among them, the ground inductive coil is inconvenient to install, and the perception range is narrow, the data blind area is large, and its instability is not suitable for timing scheme; millimeter wave radar measures the position, angle and speed of the target through the Doppler redshift principle. But millimeter wave radar cannot detect stationary objects, and cannot distinguish multiple targets existing in a small range; video analysis detects targets and target types by analyzing the video image of the optical camera through computer vision technology. However, the video analysis is limited by the resolution of the picture and the poor detection effect of the mainstream deep learning model on small targets. Long-distance targets are prone to be missed, or are greatly affected by external conditions. In the night, rain, and smog weather, the detection accuracy will also decrease, and the distance and speed of the target cannot be accurately obtained. SUMMARY

[0004] The purpose of the present application is to provide a target recognition method and system based on radar and video fusion and electronic equipment, which combines radar data and video data for road target recognition to improve the accuracy of road target recognition and improve the rationality of formulating a road adaptive timing scheme.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A target recognition method based on radar and video fusion, comprising:

[0007] Obtaining video data and radar point cloud data of a to-be-tested intersection;

[0008] Frame processing the video data to obtain a video frame sequence;

[0009] Inputting the video frame sequence into a video target recognition model to obtain a plurality of video recognition targets; the video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection;

[0010] determining a plurality of radar recognition targets according to the radar point cloud data;

[0011] fusing the plurality of video recognition targets and the plurality of radar recognition targets by using a fusion model to obtain a target recognition result of the to-be-tested intersection; the fusion model is obtained by training a multi-layer perceptron neural network by using historical video data and historical radar point cloud data of the to-be-tested intersection; and the multi-layer perceptron neural network is a dual structure.

[0012] Optionally, before the video data and the radar point cloud data of the to-be-tested intersection are obtained, the method further includes:

[0013] obtaining a plurality of historical video data of the to-be-tested intersection;

[0014] performing frame processing on the plurality of historical video data to obtain a plurality of historical video frame sequences;

[0015] annotating targets and target categories in the plurality of historical video frames in each historical video frame sequence to obtain a plurality of historical annotated video frame sequences;

[0016] training a YOLO neural network by taking the plurality of historical video frame sequences as input and taking target positions and target categories in the plurality of historical video frames as output to obtain the video target recognition model.

[0017] Optionally, the fusing the plurality of video recognition targets and the plurality of radar recognition targets by using the fusion model to obtain the target recognition result of the to-be-tested intersection includes:

[0018] determining any radar recognition target as a current radar recognition target;

[0019] inputting physical coordinate data of the current radar recognition target into the fusion model to obtain coordinate data of the current radar recognition target in a video coordinate system;

[0020] calculating, according to the coordinate data of the current radar recognition target in the video coordinate system, an Euclidean distance between the current radar recognition target and each video recognition target;

[0021] determining whether the minimum Euclidean distance is smaller than an Euclidean distance threshold to obtain a first determination result;

[0022] if the first determination result is yes, determining that the current radar recognition target is a matched radar recognition target, determining that a video recognition target corresponding to the minimum Euclidean distance is a matched video recognition target, and determining that the matched radar recognition target and the matched video recognition target are the same target;

[0023] If the first determination result is no, it is determined that the current radar recognition target is an unmatched radar recognition target.

[0024] Optionally, after determining that the current radar recognition target is a matched radar recognition target, determining that the video recognition target corresponding to the minimum Euclidean distance is a matched video recognition target, and determining that the matched radar recognition target and the matched video recognition target are the same target, the method further comprises:

[0025] obtaining coordinate data of a position of the matched video recognition target in a current video frame in a video coordinate system as first coordinates; the current video frame is any video frame other than a first video frame;

[0026] obtaining coordinate data of a position of the matched video recognition target in a previous video frame in the video coordinate system as second coordinates;

[0027] obtaining a time interval between the current video frame and the previous video frame;

[0028] inputting the first coordinates into the fusion model to obtain coordinate data of the matched video recognition target at a current time in a physical coordinate system as third coordinates;

[0029] inputting the second coordinates into the fusion model to obtain coordinate data of the matched video recognition target at a previous time in the physical coordinate system as fourth coordinates;

[0030] determining an actual moving distance of the matched video recognition target in the time interval according to the third coordinates and the fourth coordinates;

[0031] determining a moving speed of the matched video recognition target according to the time interval and the actual moving distance.

[0032] Optionally, after determining that the current radar recognition target is an unmatched radar recognition target, the method further comprises:

[0033] determining any unmatched radar recognition target as a current unmatched radar recognition target;

[0034] tracking the current unmatched radar recognition target by using a multi-target tracker, and obtaining a number of occurrences of the current unmatched radar recognition target in a preset time period after a current time;

[0035] determining whether the number of occurrences is greater than a number-of-occurrences threshold to obtain a second determination result;

[0036] if the second determination result is yes, it is determined that the current unmatched radar recognition target is a target driving out of a to-be-measured intersection;

[0037] if the second determination result is no, it is determined that the current unmatched radar recognition target is a target driving into the to-be-measured intersection.

[0038] Optionally, after determining that the current unmatched radar identification target is a target that exits the to-be-tested intersection, the method further comprises:

[0039] deleting the radar data of the target that exits the to-be-tested intersection.

[0040] Optionally, before acquiring the video data and the radar point cloud data of the to-be-tested intersection, the method further comprises:

[0041] determining, according to historical video data and historical radar point cloud data, coordinate data of the same position in a video coordinate system as first historical coordinates and coordinate data of the same position in a physical coordinate system as second historical coordinates;

[0042] training the multi-layer perceptron neural network by taking the first historical coordinates as positive inputs, taking the second historical coordinates as positive outputs, taking the second historical coordinates as negative inputs, and taking the first historical coordinates as negative outputs, to obtain the fusion model.

[0043] A target identification system based on radar and video fusion comprises:

[0044] a data acquisition module configured to acquire video data and radar point cloud data of a to-be-tested intersection;

[0045] a video frame sequence determination module configured to perform frame processing on the video data to obtain a video frame sequence;

[0046] a video identification target determination module configured to input the video frame sequence into a video target identification model to obtain a plurality of video identification targets; the video target identification model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection;

[0047] a radar identification target determination module configured to determine a plurality of radar identification targets according to the radar point cloud data;

[0048] a target identification result determination module configured to fuse the plurality of video identification targets and the plurality of radar identification targets by using a fusion model to obtain a target identification result of the to-be-tested intersection; the fusion model is obtained by training a multi-layer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection; the multi-layer perceptron neural network has a dual structure.

[0049] An electronic device comprises a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to make the electronic device execute the target identification method based on radar and video fusion according to any one of claims 1 to 7.

[0050] Optionally, the memory is a readable storage medium.

[0051] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0052] The target recognition method, system and electronic equipment based on radar and video fusion provided by the present application acquire video data and radar point cloud data of a to-be-tested intersection; the video data is subjected to frame processing to obtain a video frame sequence; the video frame sequence is input into a video target recognition model to obtain a plurality of video recognition targets; the video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection; a plurality of radar recognition targets are determined according to the radar point cloud data; a fusion model is used to fuse the plurality of video recognition targets and the plurality of radar recognition targets to obtain a target recognition result of the to-be-tested intersection; the fusion model is obtained by training a multi-layer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection; the multi-layer perceptron neural network has a dual structure. The present application improves the accuracy of road target recognition by constructing a fusion model combining radar data and video data, thereby improving the rationality of formulating a road self-adaptive timing scheme. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0054] Figure 1 The flowchart of the target recognition method based on radar and video fusion in Embodiment 1 of the present application;

[0055] Figure 2 The schematic diagram of the fusion model in Embodiment 1 of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] The purpose of the present application is to provide a target recognition method, system and electronic equipment based on radar and video fusion, which improves the accuracy of road target recognition by combining radar data and video data for road target recognition, thereby improving the rationality of formulating a road self-adaptive timing scheme.

[0058] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0059] Embodiment 1

[0060] As shown in the embodiment, a target recognition method based on radar and vision fusion is provided, comprising: Figure 1

[0061] Step 101: acquiring video data and radar point cloud data of a to-be-tested intersection;

[0062] Step 102: performing frame processing on the video data to obtain a video frame sequence;

[0063] Step 103: inputting the video frame sequence into a video target recognition model to obtain a plurality of video recognition targets; the video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection;

[0064] Step 104: determining a plurality of radar recognition targets according to the radar point cloud data;

[0065] Step 105: using a fusion model to fuse the plurality of video recognition targets and the plurality of radar recognition targets to obtain a target recognition result of the to-be-tested intersection; the fusion model is obtained by training a multi-layer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection; the multi-layer perceptron neural network is a dual structure.

[0066] For example, step 105 comprises:

[0067] Step 1051: determining any radar recognition target as a current radar recognition target;

[0068] Step 1052: inputting physical coordinate data of the current radar recognition target into the fusion model to obtain coordinate data of the current radar recognition target in a video coordinate system;

[0069] Step 1053: calculating Euclidean distances between the current radar recognition target and each video recognition target according to the coordinate data of the current radar recognition target in the video coordinate system;

[0070] Step 1054: determining whether the minimum Euclidean distance is less than a Euclidean distance threshold to obtain a first determination result; if the first determination result is yes, step 1055 is executed, and if the first determination result is no, step 1056 is executed.

[0071] ​Step 1055: determining that the current radar recognition target is a matched radar recognition target, determining that the video recognition target corresponding to the minimum Euclidean distance is a matched video recognition target, and determining that the matched radar recognition target and the matched video recognition target are the same target;

[0072] Step 1056: determining that the current radar recognition target is an unmatched radar recognition target.

[0073] After step 1055, further comprising steps 1057-10513.

[0074] Step 1057: obtaining coordinate data of a position of the matched video recognition target in a current video frame in a video coordinate system as first coordinates; the current video frame is any video frame other than the first frame video frame;

[0075] Step 1058: obtaining coordinate data of a position of the matched video recognition target in a previous video frame in a video coordinate system as second coordinates;

[0076] Step 1059: obtaining a time interval of the current video frame and the previous video frame;

[0077] Step 10510: inputting the first coordinates into the fusion model to obtain coordinate data of the matched video recognition target in a physical coordinate system at a current time as third coordinates;

[0078] Step 10511: inputting the second coordinates into the fusion model to obtain coordinate data of the matched video recognition target in the physical coordinate system at a previous time as fourth coordinates;

[0079] Step 10512: determining an actual moving distance of the matched video recognition target in the time interval according to the third coordinates and the fourth coordinates;

[0080] Step 10513: determining a moving speed of the matched video recognition target according to the time interval and the actual moving distance.

[0081] After step 1056, further comprising:

[0082] Step 10514: determining that any unmatched radar recognition target is a current unmatched radar recognition target;

[0083] Step 10515: tracking the current unmatched radar recognition target by using a multi-target tracker, and obtaining a number of occurrences of the current unmatched radar recognition target in a preset time period after a current time;

[0084] Step 10516: determining whether the number of occurrences is greater than a number-of-occurrences threshold to obtain a second determination result; if the second determination result is yes, steps 10517-10518 are executed; if the second determination result is no, step 10519 is executed.

[0085] Step 10517: If the second determination result is yes, it is determined that the current unmatched radar recognition target is a target driving out of the to-be-tested intersection.

[0086] Step 10518: Radar data of the target driving out of the to-be-tested intersection is deleted.

[0087] Step 10519: If the second determination result is no, it is determined that the current unmatched radar recognition target is a target driving into the to-be-tested intersection.

[0088] For matching of pixel plane multi-dimensional data obtained by radar data transformation and video data, matching between targets is performed based on a greedy algorithm, and a basis for determination is minimization of the Euclidean distance d of the pixel coordinates (x1, y1) after radar coordinate conversion to pixel coordinates and the pixel coordinates (x2, y2) of the video recognition result:

[0089]

[0090] For the target in the video data without matching, since a higher confidence threshold σ and an object condition category probability θ are set, the prediction accuracy of the model is considered to be reliable, and the algorithm supplements actual position, actual speed and other information of the output result according to another part of the fusion model; for the target in the radar data without matching, firstly, a tracking id is recorded for each radar target through multi-target tracking, secondly, a counter is set for the tracking id without matching, and when the current unmatched target is higher than a certain number a, the current target is not counted. The size of a is affected by the current position of the target and the set stop line parameter. In multi-target tracking, a state estimator is defined, and in consideration of historical state and observation data, the position and speed of the current target are estimated in an iterative manner to match the position and speed of the target observed in the current frame.

[0091] The target recognition method based on radar and video fusion provided in the embodiment further includes steps 106-109 before step 101.

[0092] Step 106: Obtain a plurality of historical video data of a to-be-tested intersection.

[0093] Step 107: Perform frame processing on the plurality of historical video data to obtain a plurality of historical video frame sequences.

[0094] Step 108: Label targets and target categories in a plurality of historical video frames in each historical video frame sequence to obtain a plurality of historical labeled video frame sequences.

[0095] ​Step 109: training the YOLO neural network with the plurality of historical video frame sequences as input and the target positions and target categories in the plurality of historical video frames as output to obtain a video target recognition model.

[0096] The specific video target recognition model determination method is as follows:

[0097] (1) Image acquisition and labeling: In the obtained corresponding intersection video images, the target in the image is labeled by using an image labeling tool Labelme, and the category and position of the target are labeled. The target label categories are as follows: sedan, truck, bus, engineering vehicle, off-road vehicle, van, bicycle, electric vehicle, motorcycle, pedestrian, and other targets. After labeling, the save path of the selected picture and the.xml file path of the label information are selected. The.xml file contains the labeled categories and coordinates and other information.

[0098] (2) Data set division and data enhancement: the images and label files are divided into a training set and a validation set, and the training set and the validation set account for 80% and 20%, respectively. The training data is cut into far, middle and near three parts, and different down-sampling coefficient intervals [1, 2.5], [1.5, 4] and [2, 6] are used, and the down-sampling coefficient γ of each part of the image is randomly sampled from the corresponding interval. The down-sampled image is randomly cropped to 320*320 and then input into the network.

[0099] (3) Setting network model parameters and training: in the configuration file of the YOLO network model, according to the size of the graphics card memory and the user's final recognition effect requirement, the size of the convolutional neural network input image, the number of recognition categories and the filters value, and the iteration number parameter are set. Seven anchorboxes are initialized, seven values are randomly selected in all boundingboxes as the initial values of the k anchorboxes, and the IOU value of the box and the clustering center A or B is selected as the distance index in clustering analysis:

[0100] ;

[0101] ;

[0102] The improved YOLO network structure is parameterized, the improved YOLO network structure with the set parameters is put into a computer with a configured environment, the labeled picture names in the training set and the validation set are used for training, the pictures divided in the test set are put into the computer for testing during the training process, the effect of each stage of training is obtained, the map parameter is set during the process to observe the map value of the training in real time, and the trained network model is saved after the training is completed.

[0103] (4) Pruning and quantization: the trained network model is filtered to remove some unimportant neural network connections through unstructured pruning, and then retrained for parameter fine-tuning. This process is repeated until no pruning can be performed. Further quantization is performed on the basis of pruning, and k-means clustering is selected among the weights of different layers of the network layer to obtain k intervals, and the weight parameters are distributed in the k intervals. The original weight data is replaced by the corresponding k interval values.

[0104] In addition, the target recognition method based on radar and vision fusion provided in the embodiment further includes steps 1010-1011 before step 101.

[0105] Step 1010: according to the historical video data and the historical radar point cloud data, the coordinate data of the same position in the video coordinate system is determined as the first historical coordinate, and the coordinate data in the physical coordinate system is determined as the second historical coordinate.

[0106] Step 1011: training the multi-layer perceptron neural network with the first historical coordinate as the positive input, the second historical coordinate as the positive output, the second historical coordinate as the negative input, and the first historical coordinate as the negative output, to obtain the fusion model.

[0107] As shown in Figure 2 The fusion model uses a dual MLP network to transform the target collected by the radar from the radar collection plane to the pixel plane During training, calibration can be completed by calibrating at least four non-collinear feature points. The format of the feature points is (x, y, u, v), where x and y are the actual distances with the actual position of the radar and vision fusion device as the origin, and u and v are the image coordinates with the lower left corner of the image as the origin.

[0108] The fusion model is essentially a calibration model representing the mapping relationship between the position of the target object in reality and the pixel position of its image. This part of the model has its particularity. At different intersections, the relative position and direction of the camera coordinate system and the world coordinate system are changing. The pre-trained model cannot make correct predictions under different homography relationships. Therefore, the data labeling and model training are placed on the radar-visual fusion device, and a simple and fast training framework is proposed. After completing the position coordinate pairing of the corresponding relationship, a new model that matches the current scene can be trained on the ARM chip on the device within a few minutes. The fusion model in the embodiment is trained in a cyclic framework using a dual structure. The fusion model can be regarded as two separate models, representing the transformation from the world coordinate system to the pixel coordinate system and its dual transformation. In the absence of radar data, the actual distance u, v of the target can be obtained from the target x, y coordinates output by Yolo due to the prior data provided by the video-radar model. The actual distance of the previous frame, the interval time between the previous frame and the current frame, and the actual speed of the target are calculated.

[0109] Embodiment 2

[0110] In order to perform the method corresponding to the above-mentioned embodiment one, to realize the corresponding functions and technical effects, the following provides a target recognition system based on radar-visual fusion, comprising:

[0111] A data acquisition module is configured to acquire video data and radar point cloud data of a to-be-tested intersection.

[0112] A video frame sequence determination module is configured to perform frame processing on the video data to obtain a video frame sequence.

[0113] A video recognition target determination module is configured to input the video frame sequence into a video target recognition model to obtain a plurality of video recognition targets. The video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection.

[0114] A radar recognition target determination module is configured to determine a plurality of radar recognition targets based on the radar point cloud data.

[0115] A target recognition result determination module is configured to fuse the plurality of video recognition targets and the plurality of radar recognition targets using a fusion model to obtain a target recognition result of the to-be-tested intersection. The fusion model is obtained by training a multilayer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection. The multilayer perceptron neural network has a dual structure.

[0116] The target recognition system based on radar and video fusion provided by the embodiment can analyze and adaptively fuse two kinds of data of radar and video, and obtain more accurate multi-source information. The target recognition system comprises a millimeter wave radar module, a video input and acquisition module, a signal processing module and a transmission and storage module. The video input and acquisition module comprises a complementary metal oxide semiconductor (CMOS), a video processing unit (VPU) decoding module and an encoding module. Radar data is collected and processed by the millimeter wave radar module. The signal processing module comprises a VPU, a built-in digital signal processing (DSP) chip, an advanced reduced instruction set machine (ARM) chip and an embedded neural network processing unit (NPU) chip. The DSP inputs the pictures decoded by the VPU into the NPU chip after preprocessing. The NPU runs two groups of trained deep learning models. A target detection model automatically extracts the target position and target attribute in the pictures. A radar and video fusion model converts data from a radar plane to a pixel plane. A complete adaptive radar and video fusion algorithm runs on the ARM chip, converts the fused information into a high-definition video stream, and encodes the high-definition video stream by the VPU encoding module. The high-definition video stream is encoded by H.265 (H.265-HEVC (High Efficiency Video Coding)) or MPEG4 (Moving Picture Experts Group 4) and output through a network. The transmission and storage module comprises a 5G module, an SSD module, an SD module and a video encoder. A fifth generation mobile communication technology (5G) module, a solid state disk (SSD) module and an SD (Secure Digital) module perform bidirectional signal transmission with the DSP chip through a DSP bus. The high-definition video stream is transmitted to a PC (Personal Computer) through the 5G module. The high-definition video stream is stored on an SSD solid state disk and an SD card (Secure Digital Memory Card) through the SSD module and the SD module.

[0117] The VPU, the NPU and the ARM chip realize corresponding functions through interfaces provided by the VPU, the NPU and the ARM chip respectively. The VPU adjusts the collection parameters of the CMOS module through the built-in image signal processing (ISP) image signal processor, the CMOS module collects a high-resolution image and transmits the high-resolution image to the VPU decoding module and the built-in DSP chip for image preprocessing, the NPU runs a trained deep learning model, sequentially performs target detection, target recognition and calibration of the millimeter wave radar and the optical camera in a scene, converts the license plate information into a high-definition video stream, and the VPU video encoder encodes the high-definition video stream through H.265 or MPEG4 and outputs the high-definition video stream through a network; the high-definition video stream is stored on an SSD solid state disk and an SD card through an SSD module and an SD module.

[0118] In the embodiment, the anchor box ratio of the YOLO architecture is adjusted to be close to the length-width ratio and size of vehicles and persons through clustering during training and running; different down-sampling coefficients are adopted for the distance of the road during image down-sampling; the structure of the output layer is modified, and the recognition range of target detection is adjusted; the adaptive radar and video fusion algorithm and program are used to align the same frame data of the radar and the video stream at the same time, combine the multi-source data output by the fusion model, and based on the target features, the difference and the fusion confidence, a greedy algorithm is used to determine whether it is the same target object, and a new video image containing speed, moving direction, distance and target features is output, thereby improving the recognition accuracy and precision of the target object.

[0119] Embodiment 3

[0120] The embodiment provides an electronic device, including a memory and a processor, the memory is used for storing a computer program, and the processor runs the computer program to make the electronic device execute the target recognition method based on radar and video fusion in the embodiment 1. The memory is a readable storage medium.

[0121] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the related parts can be referred to the method part.

[0122] The principles and implementation manners of the present application are described by using specific examples in this paper, and the above embodiment description is only used to help understand the method and core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A target recognition method based on radar and visual fusion, characterized in that, The method comprises the following steps: acquiring video data and radar point cloud data of a to-be-tested intersection; frame processing the video data to obtain a video frame sequence; inputting the video frame sequence into a video target recognition model to obtain a plurality of video recognition targets; the video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection; determining a plurality of radar recognition targets according to the radar point cloud data; using a fusion model to fuse the plurality of video recognition targets and the plurality of radar recognition targets to obtain a target recognition result of the to-be-tested intersection; the fusion model is obtained by training a multi-layer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection; the multi-layer perceptron neural network is a dual structure; the step of using the fusion model to fuse the plurality of video recognition targets and the plurality of radar recognition targets to obtain the target recognition result of the to-be-tested intersection comprises the following steps: determining any radar recognition target as a current radar recognition target; inputting physical coordinate data of the current radar recognition target into the fusion model to obtain coordinate data of the current radar recognition target in a video coordinate system; calculating the Euclidean distance between the current radar recognition target and each video recognition target according to the coordinate data of the current radar recognition target in the video coordinate system; judging whether the minimum Euclidean distance is less than a Euclidean distance threshold to obtain a first judgment result; if the first judgment result is yes, determining that the current radar recognition target is a matched radar recognition target, determining that the video recognition target corresponding to the minimum Euclidean distance is a matched video recognition target, and determining that the matched radar recognition target and the matched video recognition target are the same target; acquiring coordinate data of a position of the matched video recognition target in a current video frame in the video coordinate system as a first coordinate; the current video frame is any video frame other than a first video frame; acquiring coordinate data of a position of the matched video recognition target in a previous video frame in the video coordinate system as a second coordinate; acquiring a time interval between the current video frame and the previous video frame; inputting the first coordinate into the fusion model to obtain coordinate data of the matched video recognition target in the physical coordinate system at a current time as a third coordinate; inputting the second coordinate into the fusion model to obtain coordinate data of the matched video recognition target in the physical coordinate system at a previous time as a fourth coordinate; determining an actual moving distance of the matched video recognition target in the time interval according to the third coordinate and the fourth coordinate; determining a moving speed of the matched video recognition target according to the time interval and the actual moving distance; if the first judgment result is no, determining that the current radar recognition target is an unmatched radar recognition target. 2.The target recognition method based on radar and visual fusion according to claim 1, characterized in that, Before the step of acquiring the video data and the radar point cloud data of the to-be-tested intersection, the method further comprises the following steps: acquiring a plurality of historical video data of the to-be-tested intersection; frame processing the plurality of historical video data to obtain a plurality of historical video frame sequences; labeling targets and target categories in a plurality of historical video frames in each historical video frame sequence to obtain a plurality of historical labeled video frame sequences; The video target recognition model is obtained by training the YOLO neural network with the plurality of historical video frame sequences as input and target positions and target categories in the plurality of historical video frames as output. 3.The target recognition method based on radar and visual fusion according to claim 1, characterized in that, After determining that the current radar recognition target is an unmatched radar recognition target, the method further includes: determining that any unmatched radar recognition target is a current unmatched radar recognition target; tracking the current unmatched radar recognition target by using a multi-target tracker and obtaining a number of occurrences of the current unmatched radar recognition target within a preset time period after a current time; determining whether the number of occurrences is greater than a number-of-occurrences threshold to obtain a second determination result; if the second determination result is yes, determining that the current unmatched radar recognition target is a target that is driving out of the to-be-tested intersection; if the second determination result is no, determining that the current unmatched radar recognition target is a target that is driving into the to-be-tested intersection.

4. The target recognition method based on radar and visual fusion according to claim 3, characterized in that, After determining that the current unmatched radar recognition target is a target that is driving out of the to-be-tested intersection, the method further includes: deleting radar data of the target that is driving out of the to-be-tested intersection.

5. The target recognition method based on radar and visual fusion according to claim 3, characterized in that, Before obtaining the video data and the radar point cloud data of the to-be-tested intersection, the method further includes: determining, according to historical video data and historical radar point cloud data, coordinate data of a same position in a video coordinate system as first historical coordinates and coordinate data of the same position in a physical coordinate system as second historical coordinates; training the multi-layer perceptron neural network by taking the first historical coordinates as positive input, taking the second historical coordinates as positive output, taking the second historical coordinates as negative input, and taking the first historical coordinates as negative output to obtain the fusion model.

6. A target recognition system based on radar and vision fusion, characterized in that, The target recognition system based on radar and video fusion applies the target recognition method based on radar and video fusion according to any one of claims 1-5, and the target recognition system based on radar and video fusion includes: a data acquisition module configured to acquire video data and radar point cloud data of a to-be-tested intersection; a video frame sequence determination module configured to perform frame processing on the video data to obtain video frame sequences; a video recognition target determination module configured to input the video frame sequences into a video target recognition model to obtain a plurality of video recognition targets, wherein the video target recognition model is obtained by training a YOLO neural network using historical labeled video data of the to-be-tested intersection; a radar recognition target determination module configured to determine a plurality of radar recognition targets according to the radar point cloud data; a target recognition result determination module configured to fuse the plurality of video recognition targets and the plurality of radar recognition targets by using a fusion model to obtain a target recognition result of the to-be-tested intersection, wherein the fusion model is obtained by training a multi-layer perceptron neural network using historical video data and historical radar point cloud data of the to-be-tested intersection, and the multi-layer perceptron neural network has a dual structure.

7. An electronic device, comprising: The electronic device includes a memory and a processor, the memory is configured to store a computer program, and the processor is configured to run the computer program to enable the electronic device to perform the target recognition method based on radar and video fusion according to any one of claims 1-5.

8. The electronic device of claim 7, wherein, The memory is a readable storage medium.

Images