Multi-sensor instantaneous vehicle detection method, system, electronic device and storage medium

Abstract

The present invention relates to the field of smart transportation technology, and specifically to a multi-sensor instantaneous vehicle detection method, system, electronic device and storage medium. The method includes data acquisition, data processing and fusion detection. The simulated data collected by the Carla simulation platform and the real data collected by the real multi-sensor acquisition platform are processed to obtain a radar RD map and a DOA map, which are combined with the image map collected by the camera to form a corresponding vehicle data set. The three-branch convolutional neural network is first trained with the simulated data to verify the feasibility of the network. The three-branch convolutional neural network is then further trained with real data. Feature fusion and detection are performed using the three-branch convolutional neural network constructed by the present invention to obtain the detection result of the instantaneous number of vehicles. The present invention realizes the feature fusion of radar data and camera data through the three-branch convolutional neural network, and realizes the detection of the instantaneous number of vehicles, solves the occlusion problem existing in vehicle detection, and improves the accuracy of vehicle detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of intelligent transportation technology, and in particular to a multi-sensor instantaneous vehicle detection method, system, electronic device and storage medium based on three-branch convolutional neural network feature fusion. Background Art

[0002] Intelligent transportation has become a development trend, and vehicle detection is an indispensable part of it. Currently, traffic flow detection is mainly divided into video detection, millimeter wave radar detection, infrared detection, ultrasonic detection technology, and geomagnetic detection technology, depending on the type of sensor.

[0003] Among them, the video detection method is a video processing technology that detects and analyzes moving target objects on the road based on video image analysis and computer vision technology. Its disadvantage is that the accuracy is not high and it is easily affected by the environment, weather, illumination, interference, etc., and it has certain difficulties in detecting and capturing high-speed moving vehicles.

[0004] Millimeter-wave radar detection has broad application prospects in vehicle flow detection because it is not affected by weather, light, and other factors. However, millimeter-wave radar operates in the millimeter-wave frequency band and has disadvantages such as short ranging range, distance Doppler coupling, and difficulty in isolating transmitters and receivers. The measurement accuracy of millimeter-wave radar is greatly affected by weather, especially during rainfall.

[0005] The infrared detector of infrared detection is a top-mounted or roadside traffic flow detector, which consists of an infrared light-emitting tube and an infrared receiving tube. The working principle is that the modulation pulse generator generates a modulated pulse, which is radiated to the road through the infrared probe. When a vehicle passes, the infrared pulse is reflected from the vehicle body and received by the receiving tube of the probe. It is demodulated by the infrared demodulator and then triggered to output a detection signal after selection, amplification, rectification and filtering. Its disadvantage is that the dust and ice fog at the work site will affect the normal operation of the system.

[0006] Ultrasonic detection technology utilizes the reflection principle of the "Doppler effect." A transmitter emits ultrasonic waves from the top. When a vehicle passes, the receiver receives the echo at different times, allowing it to determine whether a vehicle has passed. However, this type of detection equipment has the disadvantage of requiring a top-mounted installation, which limits installation conditions. Furthermore, the sensor head has a very short lifespan when used in dusty and harsh environments like intersections.

[0007] Geomagnetic detection technology uses an inductive device, a toroidal coil, and a vehicle detector to form a tuned electronic system. When a vehicle passes or stops on the coil, the coil's inductance changes, stimulating the circuit to generate an output, thereby detecting the vehicle passing or stopping on the coil. Coil detection technology is mature and easy to master, with very accurate counting and stable performance. However, its disadvantages include limited traffic flow data, a significant impact on reliability and lifespan during the installation process, the need for repair or installation to interrupt traffic, shortening the lifespan of the pavement, and susceptibility to damage from heavy vehicles and road repairs.

[0008] In summary, the current market widely uses a single sensor for traffic flow detection, which has limitations such as environmental interference, installation condition restrictions, and high cost. Summary of the Invention

[0009] The purpose of the embodiments of the present invention is to provide a multi-sensor instantaneous vehicle detection method, system, electronic device and storage medium based on three-branch convolutional neural network feature fusion, aiming to make up for the limitations of single-sensor vehicle detection, and use radar and camera sensors for joint detection. Carla is used to simulate vehicle occlusion conditions in advance to obtain simulation data to verify the feasibility of the network, and then the occlusion problem in vehicle detection is actually solved by collecting real data in the later stage.

[0010] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions.

[0011] In a first aspect, in one embodiment provided by the present invention, a multi-sensor instantaneous vehicle detection method is provided, comprising:

[0012] Step 1: Data Collection

[0013] Simulated data collected through the Carla simulation platform and real data collected through the real multi-sensor acquisition platform;

[0014] Step 2: Data processing

[0015] The collected data is processed to obtain the radar RD map and DOA map, which together with the image map collected by the camera constitute the data set of the corresponding vehicle;

[0016] Step 3: Fusion Detection

[0017] The three-branch convolutional neural network is trained using the simulation data to verify the feasibility of the three-branch convolutional neural network, and the three-branch convolutional neural network is further trained using the real data; the feature fusion and detection of radar data and camera data are performed using the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0018] In some embodiments provided by the present invention, the Carla simulation platform is established based on a real multi-sensor acquisition platform, and the parameters of the sensors are established based on the real multi-sensor acquisition platform.

[0019] In some embodiments provided by the present invention, the Carla simulation platform generates radar point cloud data, and the radar point cloud data is processed through a mathematical formula model to obtain simulation data in the same format as the real radar original data.

[0020] In some embodiments provided by the present invention, the method for processing the simulated data and the real data is the same, and the method for processing the simulated data and the real data includes:

[0021] Acquire collected radar data, which is a three-dimensional matrix consisting of the antenna dimension, slow time dimension, and fast time dimension of the radar data.

[0022] Perform mean filtering on radar data to remove clutter from the surrounding environment and increase the signal-to-noise ratio.

[0023] Perform FFT transformation and DOA evaluation on the three-dimensional matrix of mean-filtered radar data;

[0024] The radar data is processed to generate RD graph and DOA graph, which are combined with the RGB images collected by the corresponding camera to form a training data set;

[0025] The RGB image captured by the camera is the Image image.

[0026] In some embodiments provided by the present invention, the RD graph, DOA graph and Image graph of the data set are feature extracted through three pre-trained models, and the three pre-trained models constitute branches of a three-branch convolutional neural network.

[0027] In some embodiments provided by the present invention, each sample data of the data set includes an RD map, a DOA map and an Image map.

[0028] In some embodiments provided by the present invention, the three-branch convolutional neural network inputs one sample data at a time, and each of the three images of the input sample data is subjected to preliminary feature extraction through three channels respectively. After feature extraction, each image is extracted into a 1*256 feature vector, and then the three feature vectors are spliced ​​into a 1*768 feature vector. The 1*768 feature vector is the fused data sample;

[0029] Each data sample contains information about the radar RD map, the DOA map, and the camera image map.

[0030] In a second aspect, in another embodiment provided by the present invention, a multi-sensor instantaneous vehicle detection system is provided, wherein the interactive system comprises:

[0031] The data acquisition module includes a simulation data acquisition unit and a real data acquisition unit. The data acquisition module is used to obtain simulation data collected by the Carla simulation platform and real data collected by the real multi-sensor acquisition platform;

[0032] The data processing module is used to process the collected data to obtain the radar RD map and DOA map, which together with the image map collected by the camera form the data set of the corresponding vehicle; and

[0033] The fusion detection module is used to train the three-branch convolutional neural network using the simulation data, verify the feasibility of the three-branch convolutional neural network, and further train the three-branch convolutional neural network using the real data; the feature fusion and detection of radar data and camera data are performed through the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0034] In the third aspect, in another embodiment provided by the present invention, an electronic device is provided, including: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the multi-sensor instantaneous vehicle detection method described in the first aspect above.

[0035] In a fourth aspect, in another embodiment provided by the present invention, a storage medium is provided, in which at least one executable instruction is stored, and the executable instruction enables the processor to perform operations corresponding to the multi-sensor instantaneous vehicle detection method described in the first aspect above.

[0036] Compared with the prior art, the multi-sensor instantaneous vehicle detection method, system, electronic device, and storage medium provided by the present invention utilize radar and camera sensors for joint detection, making up for the limitations of single-sensor vehicle detection. The feasibility of the constructed three-branch convolutional neural network is verified using simulated data from Carla simulation. By building an actual sensor data acquisition platform, the data is processed in the same way and then placed into the three-branch convolutional neural network for feature fusion and detection, further verifying the feasibility of the three-branch convolutional neural network fusion and improving the accuracy of the three-branch convolutional neural network for data fusion and detection. The three-branch convolutional neural network realizes the feature fusion of radar data and camera data, and realizes the detection of the instantaneous number of vehicles, solving the occlusion problem existing in vehicle detection and improving the accuracy of vehicle detection.

[0037] These and other aspects of the present invention will become more readily apparent in the following description of the embodiments. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory and are not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0039] Figure 1 The figure is a flowchart of a processing procedure of a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0040] Figure 2 The figure is an overall flow chart of data collection in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0041] Figure 3 The figure is an overall flow chart of data collection in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0042] Figure 4 Schematic diagram of the DOA branch pre-training accuracy in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0043] Figure 5 Schematic diagram of the RD branch pre-training accuracy in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0044] Figure 6 This figure is a schematic diagram of the Image branch pre-training accuracy in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0045] Figure 7 Schematic diagram of the overall operation framework of a traffic model in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0046] Figure 8 The present invention provides a flowchart for generating a radar signal graph in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0047] Figure 9 The present invention provides a flowchart for constructing a signal model for millimeter wave signals in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0048] Figure 10 Schematic diagram of time-division multiplexing transmission signals in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0049] Figure 11The present invention provides a flowchart of radar data processing in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0050] Figure 12 Schematic diagram of a three-dimensional data cube after 2DFFT of simulation data in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0051] Figure 13 The figure is a schematic diagram of camera data collection in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0052] Figure 14 This is a distance-speed RD diagram in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0053] Figure 15 This is a distance angle (DOA) diagram in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0054] Figure 16 This is an overall structural diagram of a three-branch convolutional neural network model in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0055] Figure 17 This is a schematic diagram of the accuracy of a three-branch convolutional neural network at different learning rates in a multi-sensor instantaneous vehicle detection method according to an embodiment of the present invention.

[0056] Figure 18 4 is a system block diagram of a multi-sensor instantaneous vehicle detection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Currently, traffic flow detection technologies are categorized by sensor type, including video detection, millimeter-wave radar, infrared detection, ultrasonic detection, and geomagnetic detection. However, the widespread use of a single sensor leads to limitations such as environmental interference, installation restrictions, and high costs.

[0059] To solve the above problems, the present invention provides a multi-sensor instantaneous vehicle detection method, system, electronic device and storage medium, which mainly collect sensor data and build a three-branch convolutional neural network, and use radar and camera sensors for joint detection. Carla is used to simulate vehicle occlusion conditions in the early stage to obtain simulation data to verify the feasibility of the network. Later, real data is collected to actually solve the occlusion problem in vehicle detection.

[0060] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the disclosure, application, or use of the present invention. It should be noted that, unless there is a conflict, the features of the embodiments of the present invention may be combined with one another.

[0061] Specifically, the embodiments of the present application are further described below with reference to the accompanying drawings.

[0062] like Figure 1 As shown, an embodiment of the present invention provides a multi-sensor instantaneous vehicle detection method, the method comprising the following steps:

[0063] Step 1: Data Collection

[0064] Simulated data collected through the Carla simulation platform and real data collected through the real multi-sensor acquisition platform;

[0065] Step 2: Data processing

[0066] The collected data is processed to obtain the radar RD map and DOA map, which together with the image map collected by the camera constitute the data set of the corresponding vehicle;

[0067] Step 3: Fusion Detection

[0068] The three-branch convolutional neural network is trained using the simulation data to verify the feasibility of the three-branch convolutional neural network, and the three-branch convolutional neural network is further trained using the real data; the feature fusion and detection of radar data and camera data are performed using the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0069] In this embodiment, see Figure 1 As shown, the present invention provides a multi-sensor instantaneous vehicle detection method based on three-branch convolutional neural network feature fusion, which has three main steps, namely data acquisition, data processing, and fusion detection.

[0070] In this embodiment, Carla is used as a tool to generate simulation data, primarily to verify the feasibility of the constructed three-branch convolutional neural network. At the same time, by building an actual sensor data acquisition platform, the data is processed identically and then fed into the three-branch convolutional neural network for feature fusion and detection. This further verifies the feasibility of the three-branch convolutional neural network fusion and improves the accuracy of the three-branch convolutional neural network for data fusion and detection.

[0071] See also Figure 2As shown in the figure, during data collection, the simulated data collected by the Carla simulation platform and the real data collected by the real multi-sensor acquisition platform are obtained. Among them, the Carla simulation platform is established based on the real multi-sensor acquisition platform, and the various sensor parameters are also established based on the real multi-sensor acquisition platform.

[0072] The Carla simulation platform generates radar point cloud data, which is processed by mathematical formula models to obtain simulation data with the same format as the real radar raw data. Figure 2 As shown in the figure, the Carla simulation platform collects data through millimeter-wave radar and cameras, generates point cloud data through the simulation platform, and brings the point cloud data into the corresponding mathematical formula model to obtain simulation data in the same format as the real radar original data. The generated simulation data is processed, and at the same time, the real data also goes through the same data processing flow as the simulation data.

[0073] In this embodiment, the simulation data is collected by Carla, and the real data is collected by the TI AWR1642+DCA1000 radar acquisition development board and the Hikvision DS-2CD2820F camera.

[0074] The real-world data acquisition platform primarily consists of a TI AWR1642+DCA1000 radar acquisition development board and a Hikvision DS-2CD2820F camera. After spatial and temporal calibration, the two devices combine time and space to collect traffic data. The raw data is processed in a data processing module to generate Direction of Arrival (DOA) maps, Radiated Recognition (RD) maps, and Image maps. This data is then fed into the network and trained in the same manner as simulated data. Here, simulated data is first used to verify the feasibility of the network, followed by real data for further validation. Therefore, the training dataset includes both simulated and real data.

[0075] When processing simulation data and real data, see Figure 1 and Figure 2As shown in the figure, radar data is processed to generate RD and DOA maps, which, combined with the corresponding RGB images (i.e., image maps) captured by the camera, form the training dataset. The data in the dataset is processed using a pre-trained model. Since the features of the three types of data (RD, DOA, and image) are different, different pre-trained models are used for feature extraction. Through this pre-training step, the pre-trained models are continuously improved and optimized. It was found that the pre-trained model achieved the best test accuracy when the learning rate was 0.001, the loss function was the cross-entropy loss function, and the number of iterations was 100. The cross-entropy loss function can represent the degree of difference between the true probability distribution and the predicted probability distribution of a random variable, and it has fewer local extreme points. The cross-entropy has a good convergence effect on solving the network optimal value problem.

[0076] At the same time, each improved pre-training model is an important component of the final three-branch convolutional neural network. The pre-training model only extracts data features and detects and recognizes one type of data. When the improved pre-training model reaches a certain level of accuracy in detecting and recognizing one type of data, it means that the model's feature extraction ability for the data meets the requirements for the final three-branch convolutional neural network. Figure 4 、 Figure 5 and Figure 6 As shown, it is the test accuracy of each data in the pre-training model training.

[0077] Specifically, sensor data collection involves three main steps: traffic model establishment, sensor spatiotemporal synchronization, and data processing. In this embodiment, traffic model establishment is implemented using the Carla simulation software. Developers can use the built-in Python API to autonomously control and simulate scenarios, including the construction of scenarios involving traffic participants and weather, the configuration of hazardous scenarios, and sensor configuration.

[0078] In this application, you can use the 7 maps that come with the open source software Carla as data simulation scenarios. When building a traffic model to simulate data simulation scenarios, the overall operation framework of the traffic model can be found in Figure 7 As shown in the figure, the client is used to control the server, and the server is used to feed data back to the client. The client performs model construction, parameter setting, and behavior planning, while the server performs vehicle autonomous driving simulation, pedestrian simulation, and sensor simulation.

[0079] Among them, see Figure 8 As shown, the method for reversely generating a radar signal map from radar point cloud data includes:

[0080] S201, constructing a signal model based on the millimeter wave signal of the radar, inputting the distance, speed, and angle of the collected point cloud data into the signal model to obtain simulation data;

[0081] S202, processing the simulation data to obtain preliminary screening results of the simulation data;

[0082] S203 , by comparing the preliminary screening results of the collected radar point cloud data with the generated radar signal graph, checking whether the generated radar signal graph matches the collected data, and visually comparing it with the image collected by the camera.

[0083] In this embodiment, the reverse generation of radar signal graphs is mainly divided into three steps: (1) mathematical modeling of millimeter wave signals, (2) simulation data processing, and (3) data correspondence verification.

[0084] Among them, see Figure 9 As shown in the figure, the method for constructing a signal model for millimeter wave signals is:

[0085] S2011. Generate a transmit signal based on a time-division multiplexing simulation of the radar transmit antenna, and receive and separate the echo signals from different transmit antennas at the receiving end through the receive antenna. The echo signals are mixed by the mixer to obtain an intermediate frequency signal.

[0086] S2012, a steering vector based on the signal angle composed of the target's distance, speed, delay caused by the distance to the receiving antenna, and the spatial phase difference of the signal when received by each receiving antenna;

[0087] S2013. A signal model is obtained by combining the intermediate frequency signal, the delay caused by the distance, and the steering vector of the signal angle.

[0088] In this embodiment, the millimeter wave signal is mathematically modeled, and the radar uses the AWR1642 millimeter wave radar. The AWR1642 millimeter wave radar transmit antenna has two multiplexing modes, namely time division multiplexing (TDM) and binary phase modulation (BPM), to ensure that the receiving antenna can separate the echo signals of different transmit antennas at the receiving end. This embodiment uses a time division multiplexing simulation to generate a signal. For a schematic diagram of the time division multiplexing transmit signal, see Figure 10 shown.

[0089] In this embodiment, the signal model is:

[0090]

[0091] Among them, m represents the number of chirps in the slow time dimension, n represents the number of sampling points in the fast time dimension, k represents the number of virtual antennas, and f sDenotes the fast time dimension sampling rate, and η represents Gaussian white noise. Simulation data is obtained by substituting the distance, speed, and angle of the collected point cloud data into the signal model formula.

[0092] In the embodiment of the present invention, see Figure 11 As shown, the radar data processing method is:

[0093] S2021. Obtain generated radar data, where the radar data is a three-dimensional matrix including an antenna dimension, a slow time dimension, and a fast time dimension of the radar data;

[0094] S2022. Perform mean filtering on the radar data to filter out clutter from the surrounding environment and increase the signal-to-noise ratio.

[0095] S2023. Perform FFT transformation on the three-dimensional matrix of the radar data after the mean filter, plot the three-dimensional data using Matlab, and perform slicing processing after determining the target to obtain a range and velocity map of the target;

[0096] S2024. Perform angle estimation based on the MUSIC algorithm to obtain a distance angle map.

[0097] In this embodiment, when radar data is processed, the radar data generated is actually a 3D matrix, namely antenna dimension*slow time dimension*fast time dimension. By performing FFT transformation on the 3D matrix, a preliminary screening of targets can be performed. Figure 3 As shown in FIG, the specific processing flow is as follows: mean filtering; 1D FFT; 2D FFT; DOA evaluation.

[0098] See also Figure 3 As shown, the method for processing the simulated data and the real data is the same, and the method for processing the simulated data and the real data includes:

[0099] Acquire collected radar data, which is a three-dimensional matrix consisting of the antenna dimension, slow time dimension, and fast time dimension of the radar data.

[0100] Perform mean filtering on radar data to remove clutter from the surrounding environment and increase the signal-to-noise ratio.

[0101] Perform FFT transformation and DOA evaluation on the three-dimensional matrix of mean-filtered radar data;

[0102] The radar data is processed to generate RD graph and DOA graph, which are combined with the RGB images collected by the corresponding camera to form a training data set;

[0103] The RGB image captured by the camera is the Image image.

[0104] Among them, the mean filter is mainly used to remove the clutter brought by the surrounding environment and increase the signal-to-noise ratio. After 2D FFT, the speed and distance of the target can generally be displayed in a three-dimensional cube. Here, MATLAB is used to plot the three-dimensional data, as shown in the following example: Figure 12 As shown, a bright spot can be seen, which is the target. By slicing the cube along the antenna dimension, the target's range and velocity diagram can be obtained. Then, the MUSIC algorithm is used to estimate the angle and obtain the range and angle diagram, see Figure 13 shown.

[0105] In this embodiment, data correspondence verification is also included. Data correspondence verification is mainly carried out by comparing the collected point cloud data with the generated radar signal map to see whether the generated radar signal map matches the collected data. The point cloud data contains distance, speed, and angle information. The radar signal map will also contain the distance, speed, and angle information of the corresponding target. Then, an intuitive comparison is made based on the pictures collected by the camera. After the radar and the camera are associated in time and space, the camera pictures at the corresponding time are searched to see the vehicle distance distribution. It can be compared with the target distribution in the radar signal map to see whether the distribution is consistent. Here, the data acquisition end saves data and generally collects 1 to 5 points for the object. For large vehicles, the number of points collected is generally more than that for small vehicles. See Figure 14 and Figure 15 As shown in the figure, it can be seen that the truck partially blocks the car, and the car signal has fewer bright spots in the radar signal image, while the truck has larger bright spots.

[0106] In this embodiment, each data set includes three data types: DOA (distance-angle diagram), RD (distance-speed diagram), and Image (camera-captured image). There are five scenarios for real-world vehicle operation on the road: no vehicle, one vehicle, two vehicles, three vehicles, and four vehicles.

[0107] In this example, 500 sets of data were collected for each scenario using the Carla simulation software. Each set of data included three data types: DOA (distance-angle diagram), RD (distance-velocity diagram), and Image (camera-captured image). The data collected by the simulation software was further divided into five scenarios: no vehicle, one vehicle, two vehicles, three vehicles, and four vehicles. The collected data was then categorized and placed into different folders based on file type.

[0108] Based on the prepared dataset, pre-training is first performed. The purpose of pre-training is to retain the parameters and weights with good training effects. Using the idea of ​​transfer learning, the pre-trained parameters are applied to the early training of the three-branch network in the later stage. The advantage of this is that compared with zero-based training, the initial training effect will achieve a certain accuracy. The accuracy will not start from zero, but will slowly increase from a certain value, saving time while ensuring accuracy. Currently, the more mainstream networks for image feature extraction are AlexNet, ZFNet, VGGNet, GoogLeNet, ResNet, etc. This pre-training experiment mainly uses VGGNet and ResNet as the basis. On this basis, improvements are made by adding DropOut layers and Linear layers, and cross-putting the two layers at the end of the VGGNet and ResNet networks to obtain the improved versions of VGGNet and ResNet. The final three-branch convolutional neural network is built based on the two improved networks to reduce network overfitting and make the detection results more accurate.

[0109] The following is the final three-branch convolutional neural network model. The overall structure of the three-branch convolutional neural network model can be found in Figure 16 shown.

[0110] In this embodiment, see Figure 16 As shown in the figure, the pre-training process for the three-branch convolutional neural network model is as follows: The pre-training process mainly includes image pre-processing and feature extraction of the pre-trained model. Image pre-processing mainly standardizes and normalizes the image to make the data more concentrated, facilitating feature extraction in the subsequent network. The image is then resized and some data augmentation is performed using the native transforms library in PyTorch. Finally, the data is converted into a tensor. The pre-training model adopts the concept of transfer learning. After trying different pre-training models, VGG16 and Resnet18 were found to be the most effective pre-training models. Finally, VGG16 was used to pre-train the RD and Image images, and Resnet18 was used to pre-train the DOA images. Based on the pre-training, certain network layer improvements were made, including three fully connected layers and two dropout layers to reduce network overfitting.

[0111] In the present invention, through the training of the early pre-training model, the feature extraction ability of each type of data reaches the predetermined effect, and finally each pre-training model is integrated, because each pre-training model is the basis of the three-branch convolutional neural network, and the early pre-training enables us to extract the radar RD map, DOA map and camera image. Figure 3The feature extraction of the three types of data has a certain priori basis. Now the three types of data are input into the three-branch convolutional neural network built according to the pre-training model at the same time. The three types of data are used as a basic sample. An RD image, a DOA image and an Image image are input each time. The three images are subjected to preliminary feature extraction through three channels respectively. After feature extraction, each image is converted into a 1*256 feature vector. Then, the three feature vectors are spliced ​​into a 1*768 feature vector. The 1*768 feature vector is the fused data sample. Each data sample contains the information of the radar RD image and DOA image, as well as the information of the camera Image image. By fusing radar information and camera information, we achieve our ultimate sensor fusion. Compared with the current fusion of radar point cloud and camera information, our fusion retains more information from the radar original data. The radar point cloud represents a lot of target information through a cluster of point clouds or a single point. In the process of processing the radar signal graph into a radar point cloud, a lot of target information will be lost, and it will also lead to incorrect estimation and recognition of the target. We don't have to worry about this problem when using the radar signal graph. The radar signal graph retains the original information of the target very well, and there will be no target recognition errors caused by parameter estimation errors, which reduces the complexity of data processing and increases fault tolerance.

[0112] After passing through the three channels of the three-branch convolutional neural network, the radar data and camera data have been fused into a 1*768 feature vector. Each 1*768 feature vector is a data sample for detection and recognition. All fused feature vectors are sent to the SVM for multi-class recognition and detection. By setting different learning rates, here we set the number of iterations to 200 rounds. On this basis, the model with the same parameter settings was repeated 10 times to obtain the average. The comparison shows that under the same experimental conditions, the model detection accuracy is the highest when the learning rate is 0.0001. The results are shown in the figure. Figure 17 shown.

[0113] In this embodiment, see Figure 17 As shown in the figure, a three-branch convolutional neural network is used to fuse the features of radar and camera data. By varying the learning rate, the final three-branch network test accuracy is observed. It is found that the three-branch convolutional network performs best when the learning rate is 0.0001, with the final accuracy stabilizing at around 96%. It can detect the instantaneous number of vehicles with high accuracy.

[0114] The multi-sensor instantaneous vehicle detection method of the present invention utilizes radar and camera sensors for joint detection, making up for the limitations of single-sensor vehicle detection. Real data is collected through a real multi-sensor acquisition platform, and the feasibility of the three-branch convolutional neural network is verified by simulation data. The three-branch convolutional neural network is further trained with real data. The feature fusion of radar data and camera data is achieved through the three-branch convolutional neural network, and the instantaneous number of vehicles is detected, which solves the occlusion problem in vehicle detection and improves the accuracy of vehicle detection.

[0115] In one embodiment of the present invention, see Figure 18 As shown, the present invention also discloses a multi-sensor instantaneous vehicle detection system, including a data acquisition module 100 , a data processing module 200 and a fusion detection module 300 .

[0116] The data acquisition module 100 is used to obtain simulation data collected by the Carla simulation platform and real data collected by the real multi-sensor acquisition platform.

[0117] The data processing module 200 is used to process the collected data to obtain a radar RD map and a DOA map, which together with the image map collected by the camera constitute a data set of the corresponding vehicle.

[0118] The fusion detection module 300 is used to train the three-branch convolutional neural network using the simulation data, verify the feasibility of the three-branch convolutional neural network, and further train the three-branch convolutional neural network using the real data; the feature fusion and detection of radar data and camera data are performed through the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0119] In this embodiment, the data acquisition module 100 includes a simulation data acquisition unit and a real data acquisition unit, wherein the simulation data acquisition unit is used to collect simulation data through the Carla simulation platform, and the real data acquisition unit is used to collect real data through the real multi-sensor acquisition platform.

[0120] In one embodiment, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform a multi-sensor instantaneous vehicle detection method. When the processor executes the instructions, the steps in the above-described method embodiments are implemented:

[0121] Step 1: Data Collection

[0122] Simulated data collected through the Carla simulation platform and real data collected through the real multi-sensor acquisition platform;

[0123] Step 2: Data processing

[0124] The collected data is processed to obtain the radar RD map and DOA map, which together with the image map collected by the camera constitute the data set of the corresponding vehicle;

[0125] Step 3: Fusion Detection

[0126] The three-branch convolutional neural network is trained using the simulation data to verify the feasibility of the three-branch convolutional neural network, and the three-branch convolutional neural network is further trained using the real data; the feature fusion and detection of radar data and camera data are performed using the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0127] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions for causing a computer to execute a multi-sensor instantaneous vehicle detection method, comprising the following steps:

[0128] Step 1: Data Collection

[0129] Simulated data collected through the Carla simulation platform and real data collected through the real multi-sensor acquisition platform;

[0130] Step 2: Data processing

[0131] The collected data is processed to obtain the radar RD map and DOA map, which together with the image map collected by the camera constitute the data set of the corresponding vehicle;

[0132] Step 3: Fusion Detection

[0133] The three-branch convolutional neural network is trained using the simulation data to verify the feasibility of the three-branch convolutional neural network, and the three-branch convolutional neural network is further trained using the real data; the feature fusion and detection of radar data and camera data are performed using the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles.

[0134] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program represented by computer instructions. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0135] Non-volatile memory may include read-only memory, magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory or external cache memory. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory or dynamic random access memory.

[0136] In summary, the multi-sensor instantaneous vehicle detection method, system, electronic device, and storage medium provided by the present invention utilize radar and camera sensors for joint detection, thereby overcoming the limitations of single-sensor vehicle detection. The feasibility of the constructed three-branch convolutional neural network is verified using simulated data from Carla simulation. By building an actual sensor data acquisition platform, the data is processed in the same way and then placed into a three-branch convolutional neural network for feature fusion and detection, further verifying the feasibility of the three-branch convolutional neural network fusion and improving the accuracy of the three-branch convolutional neural network for data fusion and detection. The three-branch convolutional neural network realizes the feature fusion of radar data and camera data, and realizes the detection of the instantaneous number of vehicles, thereby solving the occlusion problem existing in vehicle detection and improving the accuracy of vehicle detection.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-sensor instantaneous vehicle detection method, characterized in that: include: Step 1: Data Collection Simulated data collected through the Carla simulation platform and real data collected through the real multi-sensor acquisition platform; Step 2: Data processing The collected data is processed to obtain the radar RD map and DOA map, which together with the image map collected by the camera constitute the data set of the corresponding vehicle; Step 3: Fusion Detection The three-branch convolutional neural network is trained using the simulation data to verify the feasibility of the three-branch convolutional neural network, and the three-branch convolutional neural network is further trained using the real data; the feature fusion and detection of radar data and camera data are performed through the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles, wherein the three-branch convolutional neural network is based on the two networks of VGGNet and ResNet, and is improved on this basis by adding the DropOut layer and the Linear layer, and the DropOut layer and the Linear layer are cross-placed at the end of the two networks of VGGNet and ResNet to obtain the improved version of VGGNet and the improved version of ResNet, and the three-branch convolutional neural network is constructed based on the two improved versions of the network.

2. The multi-sensor instantaneous vehicle detection method according to claim 1, characterized in that: The Carla simulation platform is established based on a real multi-sensor acquisition platform, and the parameters of the sensors are established based on the real multi-sensor acquisition platform.

3. The multi-sensor instantaneous vehicle detection method according to claim 2, characterized in that: The Carla simulation platform generates radar point cloud data, which is processed by a mathematical formula model to obtain simulation data in the same format as the real radar raw data.

4. The multi-sensor instantaneous vehicle detection method according to claim 3, characterized in that: The method for processing the simulated data and the real data is the same, and the method for processing the simulated data and the real data includes: Acquire collected radar data, which is a three-dimensional matrix consisting of the antenna dimension, slow time dimension, and fast time dimension of the radar data. Perform mean filtering on radar data to remove clutter from the surrounding environment and increase the signal-to-noise ratio. Perform FFT transformation and DOA evaluation on the three-dimensional matrix of mean-filtered radar data; The radar data is processed to generate RD graph and DOA graph, which are combined with the RGB images collected by the corresponding camera to form a training data set; The RGB image captured by the camera is the Image image.

5. The multi-sensor instantaneous vehicle detection method according to claim 4, characterized in that: The RD graph, DOA graph and Image graph of the data set are feature extracted through three pre-trained models, and the three pre-trained models constitute branches of a three-branch convolutional neural network.

6. The multi-sensor instantaneous vehicle detection method according to claim 5, characterized in that: Each sample data of the data set includes an RD graph, a DOA graph and an Image graph.

7. The multi-sensor instantaneous vehicle detection method according to claim 6, characterized in that: The three-branch convolutional neural network inputs one sample data at a time, and each of the three pictures of the sample data is subjected to preliminary feature extraction through three channels. After feature extraction, each picture is converted into a 1*256 feature vector, and then the three feature vectors are spliced ​​into a 1*768 feature vector. The 1*768 feature vector is the fused data sample; Each data sample contains information about the radar RD map, the DOA map, and the camera image map.

8. A multi-sensor instantaneous vehicle detection system, characterized in that: include: The data acquisition module includes a simulation data acquisition unit and a real data acquisition unit. The data acquisition module is used to obtain simulation data collected by the Carla simulation platform and real data collected by the real multi-sensor acquisition platform; The data processing module is used to process the collected data to obtain the radar RD map and DOA map, which together with the image map collected by the camera form the data set of the corresponding vehicle; as well as The fusion detection module is used to train the three-branch convolutional neural network through the simulation data, verify the feasibility of the three-branch convolutional neural network, and further train the three-branch convolutional neural network using the real data; the feature fusion and detection of radar data and camera data are performed through the constructed three-branch convolutional neural network to obtain the detection result of the instantaneous number of vehicles, wherein the three-branch convolutional neural network is based on the two networks of VGGNet and ResNet, and is improved on this basis by adding the DropOut layer and the Linear layer, and the DropOut layer and the Linear layer are cross-placed at the end of the two networks of VGGNet and ResNet to obtain the improved version of VGGNet and the improved version of ResNet, and the three-branch convolutional neural network is constructed based on the two improved versions of the network.

9. An electronic device, characterized in that: It includes a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform the operation corresponding to the multi-sensor instantaneous vehicle detection method as described in any one of claims 1-7.

10. A storage medium, characterized in that: The storage medium stores at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the multi-sensor instantaneous vehicle detection method according to any one of claims 1 to 7.

Images