Radar and infrared image fusion target identification system based on deep learning

Abstract

The deep learning-based radar and infrared image fusion target recognition system comprises the following steps of: processing a millimeter wave radar point cloud picture, and converting the millimeter wave radar point cloud picture into a two-dimensional radar image; before image fusion, non-subsampled contourlet transform processing is carried out on a radar image, wavelet transform denoising processing is carried out on an infrared image, and the main purpose is to remove noise interference of the image; the radar image is subjected to constant false alarm detection, so that the radar image features are enhanced, and the recognition accuracy is improved; marking the infrared image, training by using a YOLOv12 algorithm, marking the radar image after enhancement processing, and training by using YOLOv12; and giving a weight according to the accuracy, carrying out decision-level fusion, and outputting an image. According to the method, the target identification accuracy is improved by fusing the multi-modal image and enhancing the de-noising processing.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a target recognition system based on deep learning and fusion of radar and infrared images. Background Art

[0002] In recent years, the use of sensors to identify targets has been widely used in military and other fields, and image fusion technology has developed rapidly, greatly improving the level of processing automation. However, single sensors still have inherent drawbacks in target recognition, such as strong environmental dependence: infrared recognition capabilities are reduced in meteorological conditions such as dense fog, rain, and snow that absorb infrared radiation; radar has difficulty accurately detecting and identifying targets in the presence of strong electromagnetic interference; and cameras are significantly affected by factors such as lighting and color, which can lead to recognition difficulties at night. To address the shortcomings of single-source imagery in target recognition, which is susceptible to camouflage, occlusion, contamination, and incompleteness, multi-source image fusion target recognition has been highly anticipated. This approach aims to increase target recognition accuracy by utilizing more sensors and applying appropriate target recognition theories and methods.

[0003] Multi-source image fusion refers to the integration of information from multiple images into a single, more informative or higher-quality image. Multi-source image fusion can be defined as the process of preprocessing image data from different sensor types and then using algorithms to organically combine the superior or complementary information contained in these multiple images to generate new, more complete, or more accurate, information, thereby achieving a more objective and essential understanding of the target object. This increases the information content of the fused image and makes it more effective in feature extraction, classification, target recognition, and visual perception.

[0004] Existing target recognition algorithms, both domestically and internationally, are largely mature, with representative algorithms such as SSD and YOLO. However, unprocessed, single-modal non-visible light image recognition in harsh environments will yield suboptimal results. While other approaches, both domestically and internationally, primarily fuse visible light with other modalities, this system provides a method for image fusion and target recognition using images from millimeter-wave radar and infrared thermal imaging cameras. Summary of the Invention

[0005] The purpose of this invention is to provide a target recognition system based on deep learning and radar infrared image fusion, which takes radar point cloud images and infrared images as input and outputs the fusion recognition results.

[0006] To achieve the above objectives, the present invention provides the following solutions:

[0007] A deep learning-based millimeter-wave radar and infrared thermal imaging image fusion method, comprising:

[0008] S1: Process the millimeter-wave radar point cloud image and convert it into a two-dimensional radar image;

[0009] S2: Before image fusion, the radar image is processed by non-subsampled contourlet transform (NSCT), and the infrared image is processed by wavelet transform denoising, the main purpose of which is to remove noise interference from the image;

[0010] S3: Constant false alarm rate (CFAR) is used on radar images to enhance radar image features and improve recognition accuracy.

[0011] S4: Label the infrared image and train it with the YOLOv12 algorithm. Label the enhanced radar image and train it with the YOLOv12 algorithm.

[0012] S5: Dynamically assign weights based on the accuracy of the radar image classification model and the infrared image classification model, and perform weighted decision-level fusion on the two classification results.

[0013] Furthermore, step S1 includes the following steps:

[0014] S101: Use millimeter wave radar to obtain data on pedestrians, cars, etc.;

[0015] Input: millimeter wave radar raw point cloud dataset ,in is the radial distance, is the azimuth, pitch angle, is the Doppler velocity, is the reflection intensity.

[0016] Then perform motion compensation: obtain the vehicle speed through the vehicle CAN bus , the corrected Doppler velocity is:

[0017]

[0018] S102: Spatial Mapping

[0019] Cartesian coordinate mapping layer: Convert to Cartesian coordinate system grid , resolution

[0020] S103: Generate a three-channel two-dimensional image containing intensity, height and velocity :

[0021] Furthermore, step S2 includes the following steps:

[0022] S201: NSCT processing of radar images

[0023] 1. NSCT decomposition

[0024] Input: 2D radar image ,size ;

[0025] Use non-subsampling pyramid (NSP) for multi-scale decomposition to generate low-frequency subbands and high frequency sub-band , where j represents the number of decomposition layers ( ); for high frequency sub-band , perform non-subsampled directional filter bank (NSDFB) decomposition to obtain multi-directional subband coefficients , where k represents the number of directions ( ).

[0026] 2. Coefficient enhancement

[0027] For high frequency sub-band Perform nonlinear gain adjustment to enhance edge features:

[0028]

[0029] in is the enhancement factor (preferably =0.5), is the scale-dependent normalization factor.

[0030] S202: Wavelet transform denoising of infrared images

[0031] 1. Wavelet decomposition

[0032] Input: Infrared image ,size ; Use discrete wavelet transform (DWT) to perform multi-level decomposition (such as 3 layers) to obtain low-frequency approximate coefficients and high frequency detail coefficient .

[0033] 2. Threshold denoising

[0034] For high frequency coefficients Apply improved Threshold algorithm:

[0035] ,

[0036] in is the coefficient standard deviation of subband l, noise variance Estimated through the finest-scale high-frequency subband.

[0037] Use soft threshold function to process coefficients:

[0038]

[0039] Furthermore, step S3 includes the following steps:

[0040] S301: Adaptive Constant False Alarm Rate (CFAR) Detection

[0041] 1. Threshold calculation

[0042] The ordered statistics CFAR (OS-CFAR) algorithm is used to sort the background window pixel intensities and select the kth sample as the noise level estimate. ;

[0043] The dynamic detection threshold T is determined by the following formula:

[0044]

[0045] Where α is the false alarm rate control parameter (preferably α = 3 to 5), and the preset false alarm probability (such as = ).

[0046] 2. Target judgment

[0047] If the pixel intensity at the center of the detection window , it is determined to be a target pixel, otherwise it is clutter:

[0048]

[0049] S302: Morphological Enhancement

[0050] Contour optimization: Morphological closing operation (dilation followed by erosion) is used to fill the internal holes of the target, and a 3×3 rectangular kernel is used as the structural element.

[0051] Furthermore, step S4 includes the following steps:

[0052] S401: Radar Infrared Image Annotation

[0053] Infrared image annotation: Annotate targets' visible thermal radiation signatures (e.g., human bodies, vehicles, etc.). Radar image annotation: Annotate targets' range-Doppler signatures (e.g., the direction of moving objects). Annotated data is stored in a standardized format to ensure accurate matching of infrared and radar data during training. The minimum target size threshold is set at 5×5 pixels, and blurred targets are cross-verified by three professional annotators.

[0054] S402: YOLOv12 model training

[0055] YOLOv12 is used as the basic detection framework and optimized for the characteristics of infrared and radar data:

[0056] YOLOv12 was trained using infrared and radar data to extract modality-specific features. Modality-specific enhancement methods were employed to address these characteristics, such as random noise injection and thermal radiation perturbation for infrared images, and point cloud sparsification and Doppler shift simulation for radar images.

[0057] The training ratio is 70% for the training set, 20% for the validation set, and 10% for the test set. This ratio is only an example and can be adjusted according to the data size.

[0058] Furthermore, step S5 includes the following steps:

[0059] S501: Obtain the accuracy of the radar image classification model and the infrared image classification model respectively and

[0060] S502: Calculate the dynamic weight distribution coefficient:

[0061]

[0062] S503: Radar classification results Compared with infrared classification results Perform weighted fusion: . BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is the target recognition system based on deep learning radar and infrared image fusion of the present invention

[0064] Flowchart of the process.

[0065] Figure 2 1 is a sample image in a specific implementation manner. The left side is the denoised infrared camera image, and the right side is the denoised and enhanced radar image.

[0066] Figure 3 This is the target recognition system result diagram of radar and infrared image fusion. Red means detection by both radar and infrared images, green means detection by radar only, and blue means detection by infrared only. DETAILED DESCRIPTION

[0067] The following will be combined with the accompanying drawings in the embodiments of the present invention. The multimodal dataset clearly and completely describes the technical solutions in the embodiments of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be understood as limiting the present invention.

[0068] Figure 1 This is a flow chart of a target recognition system based on deep learning and fusion of radar and infrared images according to an embodiment of the present invention. Figure 1As shown in the figure, the target recognition method based on deep learning radar and infrared image fusion includes the following steps:

[0069] S1: Process the millimeter-wave radar point cloud image and convert it into a two-dimensional radar image;

[0070] S101: Use millimeter-wave radar to obtain data on pedestrians, cars, etc.; perform motion compensation: obtain speed through the vehicle CAN bus and correct the point cloud Doppler velocity.

[0071] S102: Radar point cloud is converted to a 2D top view, and pixel intensity is mapped to [0,255]

[0072] S2: Before image fusion, the radar image is processed by non-subsampled contourlet transform and the infrared image is processed by wavelet transform denoising, the main purpose of which is to remove noise interference from the image;

[0073] S201: NSCT processing of radar images: 1. NSCT decomposition 2. Coefficient enhancement, high frequency directional subband Perform nonlinear gain adjustment to enhance edge features.

[0074] S202: Wavelet transform denoising of infrared images

[0075] The decomposition unit performs multi-level downsampling convolution on the input image to generate low-frequency components With high frequency components , where n is the number of decomposition levels.

[0076] 2.Threshold denoising: Apply improved Threshold algorithm:

[0077] ,

[0078] Use soft threshold function to process coefficients:

[0079]

[0080] The reconstructed image is grayscale stretched (normalized to 8 bits [0,255])

[0081] S3: constant false alarm detection is used on radar images;

[0082] S302:

[0083] Sliding window parameter settings

[0084] Protection unit: G = 4 (range) × 2 (Doppler);

[0085] Reference unit: N = 16 (8 units per side);

[0086] False alarm probability setting =

[0087] S303: Target judgment

[0088] If the unit to be tested I(m,n)>T, it is marked as a potential target;

[0089] Use clustering algorithm to merge adjacent detection points (Euclidean distance threshold δ = 3 distance gates)

[0090] S4: Label the infrared image and train it with the YOLOv12 algorithm. Label the enhanced radar image and train it with the YOLOv12 algorithm.

[0091] S401: Annotate the human, bicycle and car shapes in the image. The annotation format is A={( , , , , )|i=1,..., },in( , ) is the coordinate of the center of the frame, × is the frame size, ∈{1,2,3} is the category label;

[0092] S402: Use YOLOv12 as the basic detection framework and optimize it for the characteristics of infrared and radar data: Use infrared and radar data to train the YOLOv12 algorithm and extract modality-specific features. Loss function:

[0093]

[0094] Where L_box is Loss, L_obj is loss, is the classification loss;

[0095] S5: Assign weights based on accuracy and perform decision-level fusion.

[0096] S501: Obtain the accuracy of the radar image classification model and the infrared image classification model respectively and

[0097] S502: Calculate the dynamic weight distribution coefficient:

[0098]

[0099] S503: Radar classification results Compared with infrared classification results Perform weighted fusion: .

[0100] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A target recognition system based on radar infrared image fusion based on deep learning, characterized in that: include: S1. Process the millimeter-wave radar point cloud image and convert it into a two-dimensional radar image; S2. Before image fusion, the radar image is processed by non-subsampled contourlet transform (NSCT) and the infrared image is denoised by wavelet transform; S3, using constant false alarm detection (CFAR) on radar images to enhance radar image features; S4, annotate the infrared image and enhanced radar image, and use the YOLOv12 algorithm for training; S5. Dynamically assign weights based on the accuracy of the radar image classification model and the infrared image classification model, and perform weighted decision-level fusion on the two classification results.

2. The target recognition system based on deep learning and radar infrared image fusion according to claim 1 is characterized in that: Step S1 includes: S101. Use millimeter-wave radar to obtain pedestrian and vehicle target data; Input: millimeter wave radar raw point cloud dataset ,in is the radial distance, is the azimuth, pitch angle, is the Doppler velocity, is the reflection intensity; Then perform motion compensation: obtain the vehicle speed through the vehicle CAN bus , the corrected Doppler velocity is: S102. Spatial Mapping Cartesian coordinate mapping layer: Convert to Cartesian coordinate grid , resolution S103, generating a three-channel two-dimensional image including intensity, height and speed .

3. The target recognition system based on deep learning and radar infrared image fusion according to claim 1 is characterized in that: Step S2 includes: S201. NSCT processing of radar images (1) Use non-subsampling pyramid (NSP) for multi-scale decomposition to generate low-frequency subbands and high frequency sub-band , where j represents the number of decomposition layers ( ); for high frequency sub-band , perform non-subsampled directional filter bank (NSDFB) decomposition to obtain multi-directional subband coefficients , where k represents the number of directions ( ); (2) Coefficient enhancement: for high frequency direction sub-band Perform nonlinear gain adjustment to enhance edge features: in is the enhancement factor ( =0.5), is the scale-dependent normalization factor; S202, Wavelet transform denoising of infrared images (1) Wavelet decomposition: Input: infrared image ,size ; (2) Threshold denoising: for high frequency coefficients Apply improved Threshold algorithm: , in is the standard deviation of the coefficients of subband l, and the noise variance Estimation through the finest-scale high-frequency sub-band; Use soft threshold function to process coefficients: 。 4. The target recognition system based on deep learning and radar infrared image fusion according to claim 1, characterized in that: Step S3 includes: S301: Adaptive Constant False Alarm Rate (CFAR) Detection (1) Threshold calculation Use ordered statistics The algorithm sorts the background window pixel intensities and selects the kth sample as the noise level estimate; The dynamic detection threshold T is determined by the following formula: Where α is the false alarm rate control parameter (α = 3∼5), and the preset false alarm probability ( = ); (2) Target judgment If the pixel intensity at the center of the detection window , then it is determined to be a target pixel, otherwise it is clutter: S302: Morphological Enhancement Contour optimization: Morphological closing operation (dilation followed by erosion) is used to fill the internal holes of the target, and a 3×3 rectangular kernel is used as the structural element.

5. The target recognition system based on deep learning and radar infrared image fusion according to claim 1 is characterized in that: Step S4 includes: S401: Radar Infrared Image Annotation Infrared image annotation: annotate the target's visible thermal radiation characteristics; radar image annotation: annotate the target's range-Doppler characteristics; annotated data is stored in a standardized format to ensure correct matching of infrared and radar data during training; S402: YOLOv12 algorithm model training The YOLOv12 algorithm is used as the basic detection framework and optimized for the characteristics of infrared and radar data: The YOLOv12 algorithm is trained using infrared and radar data respectively to extract modal-specific features. Based on the characteristics of infrared and radar data, a modal-specific enhancement method is adopted.

6. The target recognition system based on deep learning and radar infrared image fusion according to claim 1 is characterized in that: Step S5 includes: S501: Obtain the accuracy of the radar image classification model and the infrared image classification model respectively and ; S502: Calculate the dynamic weight distribution coefficient: S503: Radar classification results Compared with infrared classification results Weighted: .

Images