An object detection method based on adversarial neural network and multi-sensor fusion

Abstract

The present invention discloses an object detection method based on a generative adversarial network and multi-sensor fusion, belonging to the field of object detection and environmental perception in driverless technology. In order to solve the problem that existing technologies cannot accurately detect obstacles at night, the method of the present invention uses a millimeter-wave radar and a vision sensor for fusion detection, and uses a generative adversarial network to train the network. The processing of millimeter-wave radar data mainly includes data clustering and preliminary screening of effective targets. The preprocessed millimeter-wave radar data and the daytime image output by the adversarial network are unified in the time and space coordinate systems, and the preprocessed millimeter-wave data and the daytime image output by the adversarial network are subtracted from the pre-sampled data of the same scene after registration to obtain the final processed data of both. Then the two data are fused to obtain the final night obstacle detection result. The present invention greatly improves the robustness of night object detection.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and environmental perception of driverless technology, and specifically refers to a sensor fusion target detection method under night conditions. Background Art

[0002] With the continuous pursuit of people for the quality of life and the continuous innovation of technology, driverless technology has been rapidly developed. As a key technology in driverless technology, the accuracy of target detection greatly determines the safety of driverless. Currently, the main sensors used in target detection technology are mainly three types, namely lidar, millimeter-wave radar, and camera. Among them, due to the relatively low price of the camera, the image recognition technology based on the camera has developed rapidly. Its principle is mainly to pre-train the model and set the target types, and then extract and analyze the features of the images collected by the camera to quickly classify and detect the targets in the images. However, there are still some fatal problems with image data, such as it is difficult to detect and classify targets under poor lighting conditions or severe overexposure. Lidar is a new development trend in recent years. Due to its strong robustness, large amount of data, and the characteristic of being unaffected by light, it has become the main development trend in the future. However, due to its relatively high price, excessive amount of data, difficulty in real-time processing, and being severely affected by rain and fog weather, it still has certain limitations in the current development. The application of millimeter-wave radar is relatively extensive. Early millimeter-wave radar was mainly applied in the military field. Now it has been widely applied in multiple fields such as driverless vehicles, drones, and intelligent transportation. The main advantages of millimeter-wave radar are strong penetrability, not being affected by dust and rain, not being affected by light, being able to work in bad weather, and being able to work all-weather. However, the main problem is its low accuracy and low resolution.

[0003] Current research on object detection mainly focuses on the situation of good lighting conditions. However, in the future development trend of driverless vehicles, it is necessary to meet the all-weather detection tasks of vehicles. Therefore, it is still necessary to detect road vehicles under poor lighting conditions such as at night. At present, there have been some new researches on night object detection. For example, Chinese invention patent number CN111965636A, with the name of "A Night Object Detection Method Based on the Fusion of Millimeter-Wave Radar and Vision", which uses a millimeter-wave radar and a vision sensor. The millimeter-wave radar is used to extract the region of interest. For the region of interest extracted by the millimeter-wave radar, the corresponding region of the image is brightened, and then the method of deep learning is used for detection and classification. Although the millimeter-wave radar and the vision sensor are used in this patent, the advantages of the two in detection under lighting conditions are not utilized. Only the millimeter-wave radar is used to extract the region of interest. Due to the accuracy problem of the millimeter-wave radar, missed detections are likely to occur. At the same time, the image of the region of interest is brightened and then detected, which is likely to cause data distortion and thus false detection. Chinese invention patent number CN106251355B, with the name of "A Detection Method for Fusing Visible Light Images and Corresponding Night Vision Infrared Images", which uses visible light images and night vision infrared images in the detection of night images. The detection method is to process the visible light image and the infrared image respectively to obtain the saliency image, and then fuse them. However, only using visible light images and night vision images cannot obtain the distance parameters between the obstacle targets, and there are still defects in the application of driverless vehicles. Summary of the Invention

[0004] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide a detection method based on an adversarial neural network and multi-sensor fusion to solve the problem that obstacles at night cannot be accurately detected in the existing technologies. The method of the present invention uses a millimeter-wave radar and a vision sensor for fusion detection, and at the same time samples road images with good lighting and no obstacles in different sections as a pre-sampling database. The method of processing the images sampled by the night vision sensor is to use an adversarial neural network. After training the network, the night images are converted and used as the input of the adversarial neural network, and the daytime images in the same scene are output. The processing of the millimeter-wave radar data includes data clustering and initial screening of effective targets. After obtaining the preprocessed data of the vision sensor and the millimeter-wave radar sensor, the output daytime pictures of the adversarial neural network are compared with the pre-sampling database of the current road section, the same scene views are screened out, and image registration is performed. Then, the preprocessed millimeter-wave radar data and the daytime images output by the adversarial network are unified in the time and space coordinate systems. The preprocessed millimeter-wave data and the daytime images output by the adversarial neural network are subtracted from the registered pre-sampled data of the same scene to obtain the final processed data of the two. Then, the data of the two are fused to obtain the final night obstacle detection result.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] The method for detecting night targets based on the adversarial neural network and multi-sensor fusion of the present invention comprises the following steps:

[0007] Step 1): Establish a pre-sampling database of obstacle-free roads under good lighting conditions;

[0008] Step 2): Sample the night driving data of the millimeter-wave radar and the vision sensor;

[0009] Further, the step 2) specifically comprises:

[0010] Step 21): Collect the millimeter-wave radar point cloud data during the night driving of the driverless vehicle, including the point cloud distribution and the distance between the point cloud and itself.

[0011] Step 22): Collect the RGB image data during the night driving of the driverless vehicle.

[0012] Step 3): Use the established database to train the adversarial neural network and the target detection network, so that the adversarial neural network can generate an output of the same-scene daytime image after inputting a night image. Enable the target detection network to identify and classify obstacles in the image.

[0013] Further, the step 3) specifically comprises:

[0014] Step 31): To train the adversarial neural network, daytime and night databases under the same scene are required. However, it is basically impossible to collect daytime and night images under the same scene. Therefore, first directly collect daytime images using the on-vehicle camera to establish a daytime dataset A, and then obtain the night same-scene dataset B by adjusting the contrast and brightness of the images in the dataset A. During the process of training the adversarial neural network model, use the daytime dataset A as the input of the real samples of the discriminator, and match its night dataset as the input of the generator. The network completes the training through mini-batch stochastic gradient descent (SGD), and sets the learning rate to 0.0002, where the mini-batch is preset to 128. Use a normal distribution with a mean of zero and a standard deviation of 0.02 as the initialization method for the weight parameters of each layer.

[0015] Step 32): The target detection network adopts the YOLOV3 network. Since the database required by the YOLOv3 network needs to be labeled, a target detection dataset C needs to be established and labeled. The images in the dataset are the images taken by the vehicle camera and some high-quality images found by the network to expand the database content. Then use VOC to make the dataset and train the network through the dataset.

[0016] Step 4): Perform clustering processing on the millimeter-wave radar sampling data.

[0017] Step 5): Compare the generated daytime image with the pre-sampling database, filter out the pre-sampling data of the same scene, and perform image registration;

[0018] Step 6): Unify the millimeter-wave radar data and the visual data in space and time, subtract the two data from the registered pre-sampling data, and then detect the data after the subtraction of the two respectively.

[0019] Further, the specific steps of the said Step 6) include:

[0020] Step 61): The space-time unification includes space synchronization and time synchronization. Among them, space synchronization is to transfer and transform the point cloud data collected by the millimeter-wave radar from the radar coordinate system to the pixel coordinate system; time synchronization is to use a pulse generator to set the trigger frequency according to the scanning frequency of the millimeter-wave radar, and obtain the millimeter-wave radar and camera data of the current frame each time it is triggered. If there is no data in the image at this moment, interpolation calculation is performed using the data of the previous and next moments.

[0021] Step 62): According to the millimeter-wave radar data and visual data after space-time unification obtained in Step 61), then use the registered pre-sampling data obtained in Step 5) as background data, subtract the millimeter-wave radar data and visual data from the pre-sampling data respectively, and use the trained yolov3 neural network to detect both.

[0022] Step 7): Perform target matching and data fusion on the detection data of both, and output the final night obstacle detection result.

[0023] Step 7): Perform target matching and data fusion on the detection data of both, and output the final night obstacle detection result.

[0024] Further, the pre-sampling database mentioned in the said Step 1) is the road pictures taken by the in-vehicle camera during the vehicle's driving on the road under good weather conditions. The vehicle speeds on expressways, first-class highways, second-class highways, third-class highways, and fourth-class highways are 100 km / h, 80 km / h, 70 km / h, 60 km / h, and 40 km / h respectively, and the sampling frequency is 30 FPS. After sampling the road data, the obstacle information in the image is removed manually, and only the road background information is retained. The obstacles are vehicles and pedestrians.

[0025] Further, the specific steps of the millimeter-wave radar clustering strategy mentioned in the said Step 4) are as follows:

[0026] Step 41): Select any point in a frame of radar scan data as the initial value of the clustering center;

[0027] Step 42): Calculate the Manhattan distance ΔR i+1 between a data point P i+1 (R i+1 , θ i ) in the radar scan data of the same frame and the clustering center P i (R i , θ i ), as well as the velocity deviation ΔV i : ΔR i = |R i+1 - R i |, ΔV i = |V i+1 - V i |.

[0028] Step 43): Compare the distance ΔR i and the velocity deviation ΔV i between the two points with the set thresholds R th , V th respectively. If it is less than the threshold, it can be determined that the two points belong to the same clustering cluster; otherwise, it is determined that the two points belong to different clustering clusters, and a new clustering is established with P i+1 (R i+1 , θ i+1 ) as the clustering center.

[0029] Step 44): If there are already multiple clustering centers, it is necessary to calculate the distance and velocity deviation ΔR, ΔV between P i+1 (R i+1 , θ i+1 ) and each clustering center in turn. If the distance and velocity deviation of this point from all clustering centers satisfy ΔR > R th , ΔV > V th , then a new clustering is established with this point as the clustering center; otherwise, it is considered that this point and the nearest clustering center belong to the same clustering cluster.

[0030] Step 45) Repeat the above steps until all the radar data points in the same frame are processed.

[0031] Furthermore, the image registration mentioned in step 5) includes three steps: key point detection and feature description, feature matching, and image transformation.

[0032] Furthermore, the visual data mentioned in step 6) is a night image captured by an in-vehicle camera and a daytime view generated by an adversarial neural network generator.

[0033] Furthermore, the target matching mentioned in step 7) includes calculating target similarity, matching targets of different sensors, and matching the same sensor with historical targets.

[0034] Furthermore, the data fusion mentioned in step 7) is a linear combination method based on weight coefficients. Since the observations of the target by the millimeter-wave radar and the camera are relatively independent, the covariance matrices of different sensors are respectively selected to weight the target parameters. The calculation method is as follows:

[0035] X ij = P j (P i + P j ) -1 X i + P i (P j + P i ) -1 X j

[0036] P = P i (P i + P j ) -1 P j

[0037] Among them, X i represents the relevant parameters of the i-th millimeter-wave radar target obtained through state estimation, and P i represents the corresponding radar covariance matrix; X j represents the relevant parameters of the j-th camera target obtained through state estimation, and P j represents the corresponding camera covariance matrix; X ij represents the target parameters obtained through fusion processing.

[0038] Advantages of the present invention:

[0039] Aiming at the problem that it is difficult to detect obstacles at night, the present invention uses a millimeter-wave radar and a vision sensor for fusion detection. At the same time, considering the poor robustness of the vision sensor in detecting obstacles at night, an adversarial neural network is adopted in this paper, and the network is trained so that when inputting night data, the generator can generate corresponding daytime images, and then fuse and detect with the millimeter-wave radar data, greatly improving the detection robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of multi-sensor fusion detection based on an adversarial neural network;

[0041] Figure 2 is a schematic diagram of the adversarial neural network principle. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings:

[0043] Embodiment

[0044] The method of the present invention uses millimeter-wave radar and visual sensors for fusion detection, and at the same time samples road images with good illumination and no obstacles in different sections as a pre-sampling database. The method for processing the images sampled by the visual sensor at night is to use an adversarial neural network. After training the network, the night images are converted and used as the input of the adversarial neural network, and the day images under the same scene are output. The processing of the millimeter-wave radar data includes data clustering and preliminary screening of effective targets. After obtaining the preprocessed data of the visual sensor and the millimeter-wave radar sensor, the output day pictures of the adversarial neural network are compared with the pre-sampling database of the current road section, the same-scene views are screened out, and image registration is performed. Then, the preprocessed millimeter-wave radar data and the day images output by the adversarial network are unified in the time and space coordinate systems, and the preprocessed millimeter-wave data and the day images output by the adversarial neural network are subtracted from the registered pre-sampled data of the same scene to obtain the final processed data of both. Then, the data of both are fused to obtain the final night obstacle detection result.

[0045] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0046] The method for night target detection based on an adversarial neural network and multi-sensor fusion of the present invention comprises the following steps:

[0047] Step 1): Establish a pre-sampling database of obstacle-free roads under good lighting conditions;

[0048] Step 2): Sample the night driving data of the millimeter-wave radar and the visual sensor;

[0049] Further, the specific steps of step 2) include:

[0050] Step 21): Collect the millimeter-wave radar point cloud data during the night driving of the driverless vehicle, including the point cloud distribution and the distance between the point cloud and itself.

[0051] Step 22): Collect the RGB image data during the night driving of the driverless vehicle.

[0052] Step 3): Use the established database to train the adversarial neural network and the target detection network, so that the adversarial neural network can generate and output day images of the same scene after inputting night images. Make the target detection network able to identify and classify obstacles in the images.

[0053] Further, the specific steps of step 3) include:

[0054] Step 31): To train the adversarial neural network, daytime and nighttime databases in the same scenario are required. However, it is basically impossible to collect daytime and nighttime images in the same scenario. Therefore, first, use the in-vehicle camera to directly collect daytime images and establish the daytime dataset A. Then, obtain the nighttime dataset B of the same scenario by adjusting the contrast and brightness of the images in dataset A. During the process of training the adversarial neural network model, use the daytime dataset A as the input of the real samples of the discriminator, and the matching nighttime dataset as the input of the generator. The network completes the training by using mini-batch stochastic gradient descent (SGD) with a learning rate set to 0.0002, where the mini-batch is preset to 128. Use a normal distribution with a mean of zero and a standard deviation of 0.02 as the initialization method for the weight parameters of each layer.

[0055] Step 32): The object detection network uses the YOLOV3 network. Since the database required by the YOLOv3 network needs to be annotated, it is necessary to establish the object detection dataset C and annotate it. The images in the dataset are the images captured by the vehicle camera and some high-quality images found by the network to expand the content of the database. Then, use VOC to make the dataset and train the network through the dataset.

[0056] Step 4): Cluster the sampling data of the millimeter-wave radar.

[0057] Step 5): Compare the generated daytime images with the pre-sampled database, filter out the pre-sampled data in the same scenario, and perform image registration;

[0058] Step 6): Unify the millimeter-wave radar data and the visual data in space and time, subtract the two data from the registered pre-sampled data, and then detect the data after subtraction for both of them respectively.

[0059] Further, the specific content of step 6) includes:

[0060] Step 61): The space-time unification includes space synchronization and time synchronization. Among them, space synchronization is to transfer and transform the point cloud data collected by the millimeter-wave radar from the radar coordinate system to the pixel coordinate system; time synchronization is to use the pulse generator to set the trigger frequency according to the scanning frequency of the millimeter-wave radar, and obtain the millimeter-wave radar and camera data of the current frame each time it is triggered. If there is no data in the image at this moment, interpolation calculation is performed using the data of the previous and next moments.

[0061] Step 62): According to the millimeter-wave radar data and visual data after space-time unification obtained in step 61), then use the registered pre-sampled data obtained in step 5) as the background data, subtract the millimeter-wave radar data and visual data from the pre-sampled data respectively, and use the trained yolov3 neural network to detect both of them.

[0062] Step 7): Perform target matching and data fusion on the two sets of detection data, and output the final night obstacle detection result.

[0063] Further, the pre-sampling database mentioned in step 1) is road pictures taken by the in-vehicle camera during the vehicle's road driving under good weather conditions. The vehicle speeds on expressways, first-class highways, second-class highways, third-class highways, and fourth-class highways are 100 km / h, 80 km / h, 70 km / h, 60 km / h, and 40 km / h respectively, and the sampling frequency is 30 FPS. After sampling the road data, the obstacle information in the image is removed manually, and only the road background information is retained. The obstacles are vehicles and pedestrians.

[0064] Further, the specific steps of the millimeter-wave radar clustering strategy mentioned in step 4) are as follows:

[0065] Step 41): Select any point in a frame of radar scan data as the initial value of the clustering center;

[0066] Step 42): Calculate the Manhattan distance △R i+1 (R i+1 , θ i+1 ) and the clustering center P i (R i , θ i ) between a certain data point P i and the speed deviation △V i : △R i = |R i+1 - R i |, △V i = |V i+1 - V i |.

[0067] Step 43): Compare the distance △R i and the speed deviation △V i between the two points with the set thresholds R th and V th respectively. If it is less than the threshold, it can be determined that these two points belong to the same clustering cluster; otherwise, it is determined that these two points belong to different clustering clusters, and P i+1 (R i+1 , θ i+1 ) is used as the clustering center to establish a new clustering.

[0068] Step 44): If there are already multiple clustering centers, it is necessary to calculate P i+1 (R i+1 , θ i+1) The distances to each cluster center and the velocity deviations △R and △V. If the distances and velocity deviations of this point to all cluster centers satisfy △R > R th , △V > V th , then a new cluster is established with this point as the cluster center; otherwise, it is considered that this point and the nearest cluster center belong to the same cluster.

[0069] Step 45): Repeat the above steps until all the radar data points in the same frame are processed.

[0070] Further, the image registration mentioned in step 5) includes three steps: key point detection and feature description, feature matching, and image transformation.

[0071] Further, the visual data mentioned in step 6) is a night image captured by an in-vehicle camera and a day view generated by an adversarial neural network generator.

[0072] Further, the target matching mentioned in step 7) includes calculating target similarity, matching targets of different sensors, and matching targets of the same sensor with historical targets.

[0073] Further, the data fusion mentioned in step 7) is a linear combination method based on weight coefficients. Since the observations of the millimeter-wave radar and the camera for the target are relatively independent, the covariance matrices of different sensors are respectively selected to weight the target parameters, and the calculation method is as follows:

[0074] X ij = P j (P i + P j ) -1 X i + P i (P j + P i ) -1 X j

[0075] P = P i (P i + P j ) -1 P j

[0076] Among them, X i represents the relevant parameters of the i-th millimeter-wave radar target obtained through state estimation, and P i represents the corresponding radar covariance matrix; X j represents the relevant parameters of the j-th camera target obtained through state estimation, and P j represents the corresponding camera covariance matrix; X ijIndicates the target parameter obtained through the fusion process.

[0077] The above embodiments are only used to illustrate the technical idea of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention in accordance with the technical idea proposed by the present invention shall be included within the protection scope of the present invention.

Claims

1. A detection method based on adversarial neural network and multi-sensor fusion, characterized in that, The method steps are as follows: Step 1: Sample road images with good illumination and no obstacles in different sections as a pre-sampling database; Step 2: Sample the night driving data of the millimeter-wave radar and the vision sensor; Step 3: Use the established database to train the adversarial neural network and the target detection network. Take the night images sampled by the night vision sensor as the input of the adversarial neural network, and output the day images in the same scene. The target detection network identifies and classifies the obstacles in the day images; Step 4: Cluster the sampled data of the millimeter-wave radar; Step 5: Compare the output day images of the adversarial neural network with the pre-sampling database of the current road section, filter out the same scene views, and perform image registration; Step 6: Unify the millimeter-wave radar data and the day images in space and time. Subtract the millimeter-wave radar data and the day images after space-time unification from the registered pre-sampled data, and then detect the subtracted millimeter-wave radar data and the subtracted day images respectively; Step 7: Perform target matching and data fusion on the subtracted millimeter-wave radar data and the subtracted day images detected in Step 6, and output the final night obstacle detection result.

2. The detection method based on adversarial neural network and multi-sensor fusion according to claim 1, wherein, The specific content of Step 2 includes: Step 21: Collect the millimeter-wave radar point cloud data during the night driving of the driverless vehicle, including the point cloud distribution and the distance between the point cloud and itself; Step 22: Collect the RGB image data during the night driving of the driverless vehicle.

3. The detection method based on adversarial neural network and multi-sensor fusion according to claim 1, characterized in that, The specific content of Step 3 includes: Step 31: To train the adversarial neural network, day and night databases in the same scene are required. However, it is basically impossible to collect day and night images in the same scene. Therefore, first directly collect day images using the on-vehicle camera to establish a day dataset A. Then, obtain the night dataset B in the same scene by adjusting the contrast and brightness of the images in dataset A. During the process of training the adversarial neural network model, use dataset A as the real sample input of the discriminator, and the matching night dataset as the input of the generator. The network completes the training through mini-batch stochastic gradient descent, and set the learning rate to 0.0002. Among them, the mini-batch is preset to 128, and the normal distribution with a mean of zero and a standard deviation of 0.02 is used as the initialization method for the weight parameters of each layer; Step 32: The target detection network uses the YOLOV3 network. Since the database required by the YOLOv3 network needs to be labeled, a target detection dataset C needs to be established and labeled. The images in the dataset are the images taken by the vehicle camera and some high-quality images found by the network to expand the database content. Then use VOC to make the dataset and train the network through the dataset.

4. The detection method based on adversarial neural network and multi-sensor fusion according to claim 1, characterized in that The specific content of Step 6 includes: Step 61: Space-time unification includes space synchronization and time synchronization. Space synchronization is to transfer and transform the point cloud data collected by the millimeter-wave radar from the radar coordinate system to the pixel coordinate system; time synchronization is to use a pulse generator to set the trigger frequency according to the scanning frequency of the millimeter-wave radar, and obtain the millimeter-wave radar and camera data of the current frame each time it is triggered. If there is no data in the image at this moment, interpolation calculation is performed using the data at the previous and next moments. Step 62: According to the millimeter-wave radar data and visual data after space-time unification obtained in Step 61, then use the registered pre-sampled data obtained in Step 5 as background data, subtract the millimeter-wave radar data and visual data from the pre-sampled data respectively, and use the trained yolov3 neural network to detect both.

5. The detection method based on adversarial neural network and multi-sensor fusion according to claim 1, characterized in that The pre-sampled database mentioned in Step 1 is the road pictures taken by the on-vehicle camera during the vehicle's driving on the road under good weather conditions. The vehicle speeds on expressways, first-class highways, second-class highways, third-class highways, and fourth-class highways are 100 km / h, 80 km / h, 70 km / h, 60 km / h, and 40 km / h respectively. The sampling frequency is 30 FPS. After sampling the road data, the obstacle information in the image is removed manually, and only the road background information is retained. The obstacles are vehicles and pedestrians.

6. The detection method based on the adversarial neural network and multi-sensor fusion according to claim 1, characterized in that The specific steps of the millimeter-wave radar clustering strategy mentioned in Step 4 are as follows: Step 41. Select any point in a frame of radar scan data as the initial value of the clustering center. Step 42. Calculate the Manhattan distance △Ri and speed deviation △Vi between a data point Pi+1(Ri+1,θi+1) and the clustering center Pi(Ri,θi) in the same frame of radar scan data: △Ri = |Ri+1 - Ri|, △Vi = |Vi+1 - Vi|. Step 43. Compare the distance △Ri and speed deviation △Vi between the two points with the set thresholds Rth and Vth respectively. If it is less than the threshold, it can be determined that the two points belong to the same clustering cluster, otherwise it is determined that the two points belong to different clustering clusters, and a new clustering is established with Pi+1(Ri+1,θi+1) as the clustering center. Step 44. If there are already multiple clustering centers, it is necessary to calculate the distance and speed deviation △R, △V between Pi+1(Ri+1,θi+1) and each clustering center in turn. If the distance and speed deviation of this point from all clustering centers satisfy △R > Rth and △V > Vth, then a new clustering is established with this point as the clustering center; otherwise, it is considered that this point and the nearest clustering center belong to the same clustering cluster. Step 45. Repeat the above steps until all the radar data points in the same frame are processed.

7. The detection method based on adversarial neural network and multi-sensor fusion according to claim 1, characterized in that The image registration mentioned in Step 5 includes three steps: key point detection and feature description, feature matching, and image transformation.

8. A detection method based on an adversarial neural network and multi-sensor fusion according to claim 1, characterized in that The target matching mentioned in Step 7 includes calculating target similarity, different sensor target matching, and same sensor and historical target matching. The data fusion mentioned in step 7 is based on a linear combination method of weight coefficients. Since the observations of the target by the millimeter-wave radar and the camera are relatively independent, the covariance matrices of different sensors are respectively selected to weight the target parameters, and the calculation method is as follows: X ij = P j (P i + P j ) -1 X i + P i (P j + P i ) -1 X j P = P i (P i + P j ) -1 P j Among them, X i represents the relevant parameters of the i-th millimeter-wave radar target obtained through state estimation, and P i represents the corresponding radar covariance matrix; X j represents the relevant parameters of the j-th camera target obtained through state estimation, and P j represents the corresponding camera covariance matrix; X ij represents the target parameters obtained through fusion processing.

Images