Low-slow small target identification method based on radar and visual data fusion

Abstract

The invention relates to a low-slow small target identification method based on radar and visual data fusion, and the method comprises the steps: S1, setting a radar system for collecting radar data, and a camera system for collecting visual data; s2, synchronizing data acquisition of a radar system and a camera system, and respectively obtaining corresponding radar point cloud data and image data; s3, after radar point cloud data obtained by the radar system and image data of the camera system are preprocessed, data fusion is carried out to obtain fused data; and S4, inputting the fusion data into a pre-trained neural network model, and carrying out detection and classification of low-slow small targets.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a method for identifying low, slow and small targets based on radar and visual data fusion. Background Art

[0002] Railway transportation occupies an important position in the fields of economy, military, ecology, etc., and ensuring the safety of railway transportation has become an indispensable part of railway construction. Among them, for railways built along lakes, monitoring of low-altitude, low-speed and small targets (referred to as low-speed and small targets, including targets with a flight altitude of 1-100 meters, a flight speed of 0.1-10 meters / second, and a scale range of 0.1-10 meters) is crucial. Low-speed and small targets, such as small drones, will cause safety accidents if they collide with trains on the railway line; even if low-speed and small targets maliciously carry explosives or interference objects, the harm they bring is even more unimaginable.

[0003] Existing technologies for detecting low, slow, and small targets near railways along lakes are usually based on radar or cameras.

[0004] Radar detection is proactive and has advantages in high accuracy, long distance and wide range in the mid- and high-altitude range in identifying moving targets, but the detail resolution is not high enough and imaging processing is not possible. Radar technology is applied in lake areas, where the influence of low-altitude clutter on the water surface is greater, making it relatively more difficult to accurately identify low, slow and small targets in this area.

[0005] However, technologies based on visual imaging, such as cameras, rely on the spectrum to obtain information about the detected target. They can obtain the details of the target (color, structural features) more accurately and can also perform imaging. However, the detection range is limited and it is impossible to accurately lock the target. It is also impossible to accurately know the direction and speed information of targets moving on the water surface in lake areas. Summary of the invention

[0006] The present invention provides a low-speed, slow, small target recognition method based on the fusion of radar and visual data. Compared with the existing solution of relying on single radar or camera data for target recognition, the method can not only ensure the accuracy of target recognition, but also obtain three-dimensional and precise imaging information of the target.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present application provides a method for identifying low, slow and small targets based on radar and visual data fusion, the method comprising:

[0009] S1, setting a radar system for collecting radar data, and a camera system for collecting visual data;

[0010] S2, synchronize the data acquisition of the radar system and the camera system to obtain the corresponding radar point cloud data and image data respectively;

[0011] S3, after pre-processing the radar point cloud data obtained by the radar system and the image data of the camera system, data fusion is performed to obtain fused data;

[0012] S4, inputs the fused data into the pre-trained neural network model to detect and classify low, slow and small targets.

[0013] In one implementation, S1 includes: keeping the structures and positions of the camera system and the radar system relatively fixed.

[0014] In one implementation, in S2, the data collection of the synchronous radar system and the camera system includes: after starting the radar system, calling the camera system to collect visual data through the LOAM algorithm.

[0015] In one implementation, the data fusion in S3 is the registration fusion of radar point cloud data and image data based on timestamps.

[0016] In one implementation, the preprocessing of radar point cloud data also includes clutter suppression processing in the point cloud data.

[0017] In one implementation, the clutter suppression process is based on a dual clutter suppression algorithm of OP-SVD-CFAR.

[0018] In one implementation, in S4, the neural network model adopts the YOLOv9 model.

[0019] In one implementation, after S2, the method further includes:

[0020] By jointly calibrating the radar system and the camera system, a joint calibration relationship between the two is obtained;

[0021] After S4, the method further includes:

[0022] After a low, slow, and small target is detected, three-dimensional fusion imaging of point clouds and video frames based on multiple algorithms is performed on the detected low, slow, and small target.

[0023] In one implementation, it includes:

[0024] First, the distortion correction of the image data collected by the camera system is implemented through the Open CV algorithm;

[0025] Based on the Shi-Tomasi corner detection algorithm, the pixel coordinates of the image corners of low, slow and small targets are identified and solved;

[0026] Then, the initial calibration values ​​between the radar system and the camera system are calculated by linear transformation of multiple sets of calibration plate point cloud corner coordinates and the corresponding image corner pixel coordinates, and the least square function of the difference between the point cloud reprojection coordinates and the image pixel coordinates is constructed; and the least square function is nonlinearly optimized by the improved LM algorithm to calculate the minimum value, and then the optimized joint calibration result is obtained;

[0027] Based on the principle of collinear equation and using the joint calibration results, the pixel coordinates of the image corresponding to the point cloud coordinates of the low, slow and small target are solved through the homography calculation method of image pixels and point cloud, and the RGB value of the image is assigned to the corresponding point cloud, finally realizing the 3D reconstruction of point cloud image fusion.

[0028] The technical solution of the present invention provides a low, slow and small target recognition method based on radar and visual data fusion. After solving the influence of clutter on radar data, the recognition accuracy of radar for "low, slow and small" targets along the lake can be significantly improved, so that the radar can capture target signals more accurately and accurately judge the position, speed and motion trajectory of the target; at the same time, the problem of low recognition accuracy of single photoelectric (camera) / radar is solved, and the advantages of multiple sensors are comprehensively utilized to achieve complementary advantages. The photoelectric sensor can provide high-resolution image information to make up for the shortcomings of radar in detail recognition, and the radar can play a role at long distances and in harsh environments, thereby improving the detection range and anti-interference ability of the target. The combination of the two can greatly improve the accuracy of target recognition and reduce the risk of misjudgment and missed judgment; it can also solve the problem of single photoelectric imaging lacking stereo data support and the problem of single radar imaging lacking pixel point data. It constructs a three-dimensional image of the target by combining point cloud data and image data, so that the appearance and structure of the target are more clearly presented, which helps operators to more intuitively understand the characteristics of the target, conduct more in-depth analysis and judgment, and provide a better data basis for subsequent target tracking, recognition and classification tasks, and provide more powerful support for decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a flow chart of an application scenario of a low, slow and small target recognition method based on radar and visual data fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0031] In view of the defects and problems of the prior art, the present application provides a low, slow and small target recognition method based on radar and visual data fusion, the method comprising:

[0032] S1, setting a radar system for collecting radar data, and a camera system for collecting visual data;

[0033] S2, synchronize the data acquisition of the radar system and the camera system to obtain the corresponding radar point cloud data and image data respectively;

[0034] S3, after pre-processing the radar point cloud data obtained by the radar system and the image data of the camera system, data fusion is performed to obtain fused data;

[0035] S4, inputs the fused data into the pre-trained neural network model to detect and classify low, slow and small targets.

[0036] The method provided above integrates the data collected by the radar system and the camera system, inputs the data into the neural network model, and detects and classifies low, slow and small targets. Compared with the existing single radar or single camera sensor solutions, the accuracy of target detection is improved.

[0037] The above method flow is described in more detail in the following embodiments in conjunction with the accompanying drawings of the present invention.

[0038] The method provided in this application includes the following specific steps:

[0039] (1) A radar system for collecting radar data and a camera system for collecting visual data are provided;

[0040] Specifically, the structures and positions of the camera system and the radar system are kept relatively fixed.

[0041] (2) Synchronize the data acquisition of the radar system and the camera system to obtain the corresponding radar point cloud data and image data respectively;

[0042] Specifically, after the radar system is started, the camera system is called through the LOAM algorithm to collect visual data.

[0043] The radar system integrates the LOAM (Lidar Odometry and Mapping) algorithm module, which is a SLAM (Simultaneous Localization and Mapping) algorithm based on Lidar. The algorithm is mainly used for real-time positioning and map construction, and performs odometer calculation and map construction through Lidar data.

[0044] A module for controlling the camera system is embedded in the LOAM algorithm module, so that the camera system can be controlled. The control signaling of the camera system can be transmitted via optical fiber, WiFi or Bluetooth, thereby realizing soft synchronization of data acquisition between the radar system and the camera system.

[0045] (3) By jointly calibrating the radar system and the camera system, a joint calibration relationship between the two is obtained.

[0046] Specifically, through joint calibration, the joint calibration relationship between the radar and motion camera sensors is obtained, and then accurate external parameter calibration parameters are obtained, which lays a solid foundation for the subsequent data fusion between the two sensors.

[0047] (4) After preprocessing the radar point cloud data obtained by the radar system and the image data of the camera system, data fusion is performed to obtain fused data;

[0048] Preprocessing includes preprocessing of point cloud data and image data.

[0049] In the embodiment of the present invention, considering that the influence of clutter on the water surface in lake areas cannot be ignored, the preprocessing of radar point cloud data also includes clutter suppression processing in the point cloud data.

[0050] The embodiment of the present invention uses clutter suppression to improve the signal-to-clutter ratio of detection from the perspective of clutter suppression, and detects the target on the basis of obtaining a high signal-to-clutter ratio, so that the difficulty of detection is greatly reduced, which is conducive to improving the detection performance of the radar. This project adopts a clutter suppression method based on SVD in projection space. The method first constructs a clutter space using reference unit samples in the received echo, and uses the principle of orthogonal projection (OP-CFAR, orthogonalprojection-constant false alarm rate) to calculate the orthogonal projection operator that suppresses clutter to achieve the first suppression of clutter, and then performs SVD (Singular Value Decomposition) on the projection operator, and designs a double clutter suppression operator in the signal subspace to achieve the second suppression of clutter.

[0051] (5) Input the fused data into the pre-trained neural network model to detect and classify low, slow and small targets;

[0052] This step adopts the data fusion method of point cloud and image detection based on YOLOv9.

[0053] Using deep learning methods to perform data fusion processing of point cloud and image detection can improve the recognition accuracy and real-time performance of low, slow and small targets on the water surface, and effectively solve the problem of unclear target features in target recognition. By comprehensively utilizing the information of point cloud and image, the recognition of the target can be enhanced, the misjudgment rate can be reduced, and the recognition efficiency can be improved. The YOLOv9 deep learning algorithm is used to perform target detection on the input image and point cloud respectively, and the image and detection frame, point cloud and its detection frame are fused. The correspondence between the point cloud and image detection results is determined according to the sensor calibration results, and the detection frame with higher confidence is selected as the final fusion result for output display.

[0054] (6) After a low, slow, and small target is detected, three-dimensional fusion imaging of the point cloud and video frame based on multiple algorithms is performed on the detected low, slow, and small target according to the joint calibration relationship.

[0055] Firstly, the distortion correction of the image data collected by the motion camera is realized by the Open CV optimization algorithm, and the image corner coordinates are identified and solved based on the Shi-Tomasi corner detection algorithm. Then, the initial calibration value between the two sensors is calculated by the direct linear transformation method, and the least square function of the difference between the point cloud reprojection coordinates and the image pixel coordinates is constructed. At the same time, the LM algorithm is improved and optimized by introducing the damping factor and reducing the reference value. The improved LM algorithm is used to perform nonlinear optimization on the least square function to calculate the minimum value, and then the optimized joint calibration result is obtained. Finally, the calibration result is used in the data fusion system based on the principle of collinear equation, and the pixel coordinates of the image corresponding to the point cloud coordinates are solved by the homography calculation method of image pixel points and point cloud, and the RGB value of the image is assigned to the corresponding point cloud, finally realizing the three-dimensional reconstruction of point cloud image fusion.

[0056] In summary, the technical solution of the present invention provides a low, slow and small target recognition method based on radar and visual data fusion. After solving the influence of clutter on radar data, the radar can significantly improve the recognition accuracy of "low, slow and small" targets along the lake, so that the radar can capture the target signal more accurately and accurately judge the position, speed and motion trajectory of the target; at the same time, the problem of low recognition accuracy of a single photoelectric (camera) / radar is solved, and the advantages of multiple sensors are comprehensively utilized to achieve complementary advantages. The photoelectric sensor can provide high-resolution image information to make up for the shortcomings of the radar in detail recognition, and the radar can play a role at long distances and in harsh environments, thereby improving the detection range and anti-interference ability of the target. The combination of the two can greatly improve the accuracy of target recognition and reduce the risk of misjudgment and missed judgment; it can also solve the problem of single photoelectric imaging lacking stereo data support and the problem of single radar imaging lacking pixel data. It constructs a three-dimensional image of the target by combining point cloud data and image data, so that the appearance and structure of the target are more clearly presented, which helps the operator to understand the characteristics of the target more intuitively, conduct more in-depth analysis and judgment, and provide a better data basis for subsequent target tracking, recognition and classification tasks, and provide more powerful support for decision-making.

[0057] In the several embodiments provided by the present invention, it should be understood that the disclosed method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0058] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the above-mentioned method of each embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., which can store program codes.

[0059] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A low, slow and small target recognition method based on radar and visual data fusion, characterized in that: The method comprises: S1, setting a radar system for collecting radar point cloud data, and a camera system for collecting image data; S2, synchronize the data acquisition of the radar system and the camera system to obtain the corresponding radar point cloud data and image data respectively; S3, after pre-processing the radar point cloud data obtained by the radar system and the image data of the camera system, data fusion is performed to obtain fused data; S4, inputs the fused data into the pre-trained neural network model to detect and classify low, slow and small targets.

2. The low, slow and small target recognition method based on radar and visual data fusion according to claim 1 is characterized in that: The S1 includes: keeping the structures and positions of the camera system and the radar system relatively fixed.

3. The low, slow and small target recognition method based on radar and visual data fusion according to claim 1 is characterized in that: In S2, the data collection of the synchronous radar system and the camera system includes: after starting the radar system, calling the camera system to collect visual data through the LOAM algorithm.

4. The low, slow and small target recognition method based on radar and visual data fusion according to claim 1 is characterized in that: The data fusion in S3 is the registration fusion of radar point cloud data and image data based on timestamps.

5. The low, slow and small target recognition method based on radar and visual data fusion according to claim 4 is characterized in that: The preprocessing of radar point cloud data also includes clutter suppression processing in the point cloud data.

6. The low, slow and small target recognition method based on radar and visual data fusion according to claim 5 is characterized in that: The clutter suppression process is based on a dual clutter suppression algorithm of OP-SVD-CFAR.

7. The low, slow and small target recognition method based on radar and visual data fusion according to claim 1 is characterized in that: In S4, the neural network model adopts the YOLOv9 model.

8. The low, slow and small target recognition method based on radar and visual data fusion according to claim 1 is characterized in that: After S2, the method further includes: By jointly calibrating the radar system and the camera system, a joint calibration relationship between the two is obtained; After S4, the method further includes: After a low, slow, and small target is detected, three-dimensional fusion imaging of point clouds and video frames based on multiple algorithms is performed on the detected low, slow, and small target.

9. The low, slow and small target recognition method based on radar and visual data fusion according to claim 8 is characterized in that: include: First, the distortion correction of the image data collected by the camera system is implemented through the Open CV algorithm; Based on the Shi-Tomasi corner detection algorithm, the pixel coordinates of the image corners of low, slow and small targets are identified and solved; Then, the initial calibration values ​​between the radar system and the camera system are calculated by linear transformation of multiple sets of calibration plate point cloud corner coordinates and the corresponding image corner pixel coordinates, and the least square function of the difference between the point cloud reprojection coordinates and the image pixel coordinates is constructed; and the least square function is nonlinearly optimized by the improved LM algorithm to calculate the minimum value, and then the optimized joint calibration result is obtained; Based on the principle of collinear equation and using the joint calibration results, the pixel coordinates of the image corresponding to the point cloud coordinates of the low, slow and small target are solved through the homography calculation method of image pixels and point cloud, and the RGB value of the image is assigned to the corresponding point cloud, finally realizing the 3D reconstruction of point cloud image fusion.

Images