Millimeter wave radar multi-target detection method and system based on deep learning

Abstract

The invention discloses a millimeter-wave radar multi-target detection method and system based on deep learning, and the method comprises the steps: collecting echo data of a plurality of targets in a complex environment through a millimeter-wave radar, carrying out the processing of the echo data, obtaining a distance-angle heat map, and obtaining a heat map data set; dividing the obtained heat map data set into a training set and a test set in proportion; the YOLOv8 network is improved and optimized; training and testing the optimized YOLOv8 network through the training set and the test set, and taking the trained YOLOv8 network as a target detection model; and obtaining a multi-target detection result through the target detection model. According to the method, the data collected by the millimeter wave radar is converted into the distance-angle heat map, the convolutional neural network is combined, multiple outdoor targets are detected, and the method has the advantages of being good in anti-interference performance, high in recognition accuracy and the like.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and relates to multi-target detection technology, and specifically to a millimeter-wave radar multi-target detection method and system based on deep learning. Background Art

[0002] Harsh environments, rain, snow, fog, and insufficient lighting can severely impact the accuracy of traditional visual target detection, potentially causing serious harm in applications such as indoor and outdoor surveillance and intelligent security. Among existing multi-target detection technologies, camera-based visual detection methods are widely used, but their effectiveness is limited in complex environments such as insufficient lighting and inclement weather. Millimeter-wave radar technology can address these shortcomings, but traditional millimeter-wave radar target detection methods primarily rely on the Constant False Alarm Rate (CFAR) algorithm, which processes radar signals and dynamically calculates thresholds to adapt to complex clutter environments, significantly improving target detection probability. However, in non-uniform environments, the CFAR algorithm's detection performance rapidly degrades, making it difficult to effectively address complex recognition tasks and obtain accurate solutions.

[0003] With breakthroughs in machine learning technology and the increased computing power brought about by hardware reforms, radar-collected data can be optimized through image processing, providing new research ideas for radar signal processing. For example, combining convolutional neural networks (CNNs) with millimeter-wave radar can effectively improve target detection accuracy. However, millimeter-wave radar heat maps have unique characteristics. For example, when small targets or multiple targets are present simultaneously, detection accuracy often fails to reach ideal levels. Furthermore, while the detection accuracy of multi-target detection using a Transformer-based network model is ideal, the large number of model parameters makes efficient deployment difficult on resource-constrained embedded devices. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, a millimeter-wave radar multi-target detection method and system based on deep learning is provided. The data collected by the millimeter-wave radar is converted into a distance-angle heat map, and combined with a convolutional neural network, the detection of multiple outdoor targets is realized. It has many advantages such as good anti-interference performance and high recognition accuracy. It can significantly improve the detection accuracy while maintaining a low number of model parameters, thereby laying a solid foundation for subsequent deployment on embedded devices.

[0005] Technical solution: To achieve the above objectives, the present invention provides a millimeter-wave radar multi-target detection method based on deep learning, comprising the following steps:

[0006] S1: Use millimeter-wave radar to collect echo data of multiple targets in a complex environment, process the echo data to obtain a range-angle heat map, and obtain a heat map dataset;

[0007] S2: Divide the obtained heat map dataset into training set and test set in proportion;

[0008] S3: Improve and optimize the YOLOv8 network;

[0009] S4: Train and test the optimized YOLOv8 network using the training set and test set respectively, and use the trained YOLOv8 network as the target detection model;

[0010] S5: Obtain multi-target detection results through the target detection model.

[0011] Furthermore, the millimeter wave radar transmission signal in step S1 is expressed as:

[0012]

[0013] Where ω1(t)=2πf1(t) and f1(t) is the transmission frequency, A1 is the amplitude of the transmission signal, is the phase of the transmitted signal;

[0014] The signal received by the millimeter wave radar is shown as follows:

[0015]

[0016] Where ω2(t)=2πf2(t) and f2(t) is the receiving frequency, A2 is the amplitude of the received signal, is the phase of the received signal;

[0017] The intermediate frequency signal is obtained by mixing the received signal with the transmitted signal. The intermediate frequency signal is expressed as:

[0018]

[0019] The signal is low-pass filtered and expressed as follows:

[0020]

[0021] Apply FFT on top of the low-pass filtering to determine the frequency changes:

[0022]

[0023] Once the frequency change is determined, the distance is calculated using the following equation:

[0024]

[0025] Where c is the speed of light, the time it takes for the chirp to go from minimum frequency to maximum frequency, and B is the bandwidth.

[0026] Furthermore, in step S1, the millimeter wave radar includes two antennas, the first antenna measures the distance R; assuming the target is far enough away, it can be assumed that the second antenna will measure the distance R'=R+psinθ, where p is the distance between the two receiving antennas and θ is the angle of the target relative to the radar;

[0027] The expression for θ is:

[0028]

[0029] Having determined R and θ, convert from polar coordinates to Cartesian coordinates:

[0030] Furthermore, the improvement and optimization of the YOLOv8 network in step S3 includes optimization of the backbone network part, optimization of the feature fusion part and optimization of upsampling.

[0031] Furthermore, the optimization of the backbone network in step S3 includes:

[0032] The SimSPPF module is introduced to replace the SPPF module. The SimSPPF module includes a series of convolution operations and multiple maximum pooling and splicing operations. The convolution operation includes an initial SimConv convolution layer for preliminary processing of the input feature map, reducing the number of channels to half of the original. SimConv is a custom convolution module, including convolution operation, batch normalization, and ReLU activation function. Its function is to perform convolution operation on the input feature map to extract features, and accelerate the training process and improve the stability of the model through batch normalization; the ReLU activation function introduces nonlinearity to enhance the expressive power of the model; the module downsamples the feature map through multiple maximum pooling operations to capture feature information of different scales. The feature map after each maximum pooling will be spliced with the original feature map to achieve multi-scale feature fusion; finally, a SimConv convolution layer is used to convert the fused feature map into the specified number of output channels.

[0033] Furthermore, the optimization of the feature fusion part in step S3 includes:

[0034] BiFPN is used for multi-scale feature fusion. BiFPN achieves more comprehensive information transmission and fusion through top-down and bottom-up bidirectional paths.

[0035] Furthermore, the BiFPN in step S3 includes a bidirectional cross-scale connection and a weighted feature fusion mechanism, specifically:

[0036] BiFPN simplifies the network structure and enhances information flow by optimizing cross-scale connections, removing nodes with only one input, and adding additional connections between input and output nodes at the same level. In addition, BiFPN treats each bidirectional path as a feature network layer and repeats the layer multiple times to achieve higher-level feature fusion.

[0037] During the feature fusion process, BiFPN introduces learnable weights, allowing the model to adaptively adjust the fusion method based on the importance of the features. Specifically, BiFPN uses a fast normalized fusion method to optimize the fusion results by assigning learnable weights to each input feature and dynamically adjusting these weights during training. This method not only avoids the distortion of feature information in traditional fusion methods, but also improves the model's adaptability to features of different scales.

[0038] BiFPN uses neural network architecture search technology to find the optimal feature network topology; this design enables BiFPN to better integrate multi-scale features while maintaining efficient computing performance, significantly improving the model's feature expression ability and detection accuracy.

[0039] The top-down path formula in BiFPN is:

[0040]

[0041] in, is the intermediate feature of the lth layer in the top-down path, is the input feature of the lth layer, Resize is the upsampling or downsampling operation for resolution matching, Conv is the convolution operation, w1 and w2 are learnable weights, and ∈ is a small constant for numerical stability.

[0042] The bottom-up path in BiFPN is expressed as:

[0043]

[0044] in, is the output feature of the lth layer in the bottom-up path, is the intermediate feature of the top-down path, Resize is the operation for resolution matching, and w1′, w2′, and w3′ are learnable weights.

[0045] Furthermore, the optimization of upsampling in step S3 includes:

[0046] Introducing dynamic upsampling to enhance the model's ability to recover feature maps;

[0047] Dysample is introduced to enhance the feature recovery capability, and an upsampling strategy is designed from the perspective of dynamic point sampling. The feature map is reorganized by dynamically generating sampling points. Specifically, Dysample converts the low-resolution feature map χ∈R H×W×C As input, the dynamic range adjustment factor δ is first generated by the linear transformation layer Linear1, and the value range is limited to [0, 0.5] to control the sampling range; then, another linear transformation layer Linear2 is used to generate the initial offset By combining the dynamic adjustment factor δ with the initial offset O init , generating dynamic offset O dya , the specific expression is:

[0048] O dya =O init ·δ

[0049] Then, the dynamic offset O dya Added to the fixed grid position G, a dynamic sampler S∈R is generated SH×SW×2 , define the new two-dimensional coordinates of all sampling points:

[0050] S=G+O dya

[0051] The dynamic sampler can adaptively adjust the distribution of sampling points to more accurately cover the semantic features of the target area. Based on the sampler S, the input feature map χ is reorganized by grid sampling to generate a high-resolution feature map χ′∈R SH×SW×C :

[0052] χ′=gridsample(χ, S)

[0053] The dynamic sampling mechanism makes the feature map reconstruction process more efficient by dynamically adjusting the position of the sampling points. In addition, Dysample does not rely on the generation of dynamic convolution kernels, but directly completes feature reconstruction based on the point sampling strategy, further reducing computational redundancy. Its complexity is expressed as:

[0054] T dysample =H·W·S 2 ·C

[0055] Among them, H and W represent the height and width of the input feature map respectively; S is the upsampling ratio; C is the number of channels.

[0056] Furthermore, the dynamic upsampling process in step S3 includes: first, the technology reads the original data, analyzes the data distribution, selects the basic sampling method according to the task requirements, and initializes the parameters; then, uses the initial sampling strategy to generate a training batch, inputs it into the model for training, and tracks the model performance (such as loss function, accuracy, recall rate, etc.) or sample difficulty in real time; dynamically calculates the priority or weight of the sample based on the monitoring results, for example, samples the minority class samples, and adjusts the sampling probability or weight in real time to generate a new training round; finally, repeats the training, monitoring, and adjustment of the sampling process to gradually optimize the model.

[0057] The present invention also provides a millimeter wave radar multi-target detection system based on deep learning, including a peripheral interface, a memory and a processor;

[0058] The peripheral interface is used to implement data input and output during data interaction with various external devices, thereby ensuring effective connection and communication between the system and the external devices;

[0059] The memory is configured to store computer program instructions that can be executed on the processor;

[0060] The processor is used to execute the steps of the millimeter wave radar multi-target detection method based on deep learning when running the computer program instructions.

[0061] YOLOv8's network architecture consists of three main parts: the backbone network, the neck network, and the detection head. The backbone network is responsible for extracting multi-scale features from the input image. The core modules include the C2f module, the convolution module, and the pyramid pooling module.

[0062] In the backbone network, YOLOv8 replaces the C3 module from YOLOv5, using a more efficient BottleneckBlock and residual connection design to reduce parameter count and enhance feature extraction capabilities. Its architecture includes a Split operation and stacking of multiple Bottleneck blocks, supporting dynamic gradient flow. The convolutional module, consisting of Conv2d, BatchNorm2d, and SiLU activation functions, supports downsampling and nonlinear feature mapping. Pyramid pooling, through cascaded max-pooling layers, achieves multi-scale feature fusion, significantly improving the receptive field and reducing computational redundancy.

[0063] Upsampling is used to gradually restore feature resolution, so Dysample is introduced to enhance feature recovery capabilities. The feature fusion part is responsible for fusing features at different levels and optimizing multi-scale detection capabilities. Early detectors usually make predictions directly based on the pyramid feature hierarchy extracted from the backbone network. Feature Pyramid Network (FPN) proposes a top-down approach to combine multi-scale features. Based on FPN, PANet adds an additional bottom-up path aggregation network, and then BiFPN introduces learnable weights to learn the importance of different input features, while repeatedly applying top-down and careful multi-scale feature fusion to enhance the combination of semantic information and positioning information.

[0064] The head (detection head) completes target classification and positioning detection. The decoupling head separates the classification and regression tasks to reduce task interference and improve accuracy. The Anchor-Free mechanism is used to directly predict the target center point, simplify the training process, avoid anchor box hyperparameter tuning, and use Task-AlignedAssigner to dynamically allocate positive and negative samples. Combined with Distribution FocalLoss and CIoU Loss, it optimizes classification and positioning accuracy.

[0065] During model training, the required millimeter-wave radar heat map dataset is first prepared. The model size is selected according to the task. The configuration file is used to define the network structure of the backbone network, neck network, and detection head. The most appropriate loss function and training strategy are selected, and the effectiveness of the algorithm is verified through model training. The main detection process is roughly as follows: First, the image size is adjusted to an appropriate size (such as 640*640), normalized, and the channel order is converted; second, forward propagation is performed, the input image is extracted through the Backbone part, and the Neck part fuses multi-scale information. The detection head outputs the category, confidence, and coordinates of the predicted box; next, post-processing is performed to filter low-confidence prediction boxes, remove overlapping boxes through non-maximum suppression, and retain the optimal detection result; finally, the detection box is superimposed on the original image, and the category label, coordinates, and confidence are output.

[0066] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0067] 1. It can enhance the ability to extract millimeter-wave radar thermal map features, enrich the details and diversity of model feature representation through a two-way information transmission mechanism, and effectively improve feature fusion capabilities.

[0068] 2. It can effectively improve the detection of multiple targets in complex environments. Through the combination of millimeter-wave radar sensors and neural networks, it makes up for the shortcomings of visual sensors in harsh environments, rainy, snowy, foggy weather and insufficient light conditions. It has strong anti-interference capabilities and high recognition accuracy. It can effectively improve the application accuracy and efficiency of millimeter-wave radar in multi-target detection.

[0069] 3. The present invention can significantly improve detection accuracy while maintaining a low number of model parameters, laying a solid foundation for subsequent deployment on embedded devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the angle of the target relative to the radar;

[0071] Figure 2 This is the SimSPPF structure diagram;

[0072] Figure 3 This is the BiFPN structure diagram;

[0073] Figure 4 This is the Dysample structure diagram;

[0074] Figure 5 This is the network structure diagram of the improved YOLOv8 model;

[0075] Figure 6 Comparison chart of mAP@0.5 of the model before and after improvement;

[0076] Figure 7 Comparison chart of model accuracy before and after improvement;

[0077] Figure 8 It is a confusion matrix diagram. DETAILED DESCRIPTION

[0078] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0079] Example 1:

[0080] This embodiment provides a millimeter-wave radar multi-target detection method based on deep learning, including the following steps:

[0081] S1: Use millimeter-wave radar to collect echo data of multiple targets in a complex environment, process the echo data to obtain a range-angle heat map, and obtain a heat map dataset;

[0082] This embodiment uses an FMCW millimeter-wave radar to collect echo data from multiple targets. The FMCW millimeter-wave radar consists of a transmitting antenna (RX), a receiving antenna (TX), a radio frequency signal module, a mixer, and an analog-to-digital converter (ADC).

[0083] FMCW millimeter-wave radar gradually changes the frequency of its signal using an increasing ramp. The synthesizer generates a variable-frequency signal that varies with a sawtooth wave function. This signal is amplified using a power amplifier (PA) and transmitted using one or more antennas. When reflected by an object, it is received by the receiving antenna and amplified by the low-noise amplifier (LNA). The transmitted and received signals are mixed to produce an intermediate frequency (IF) signal, which is then filtered using a low-pass filter. An analog-to-digital converter samples the IF signal and stores it in a buffer. A digital signal processor restores the buffer's contents to apply signal processing algorithms.

[0084] The millimeter wave radar transmission signal is expressed as:

[0085]

[0086] Where ω1(t)=2πf1(t) and f1(t) is the transmission frequency, A1 is the amplitude of the transmission signal, is the phase of the transmitted signal;

[0087] The signal received by the millimeter wave radar is shown as follows:

[0088]

[0089] Where ω2(t)=2πf2(t) and f2(t) is the receiving frequency, A2 is the amplitude of the received signal, is the phase of the received signal;

[0090] The intermediate frequency signal is obtained by mixing the received signal with the transmitted signal. The intermediate frequency signal is expressed as:

[0091]

[0092] The signal is low-pass filtered and expressed as follows:

[0093]

[0094] Apply FFT on top of the low-pass filtering to determine the frequency changes:

[0095]

[0096] Once the frequency change is determined, the distance is calculated using the following equation:

[0097]

[0098] Where c is the speed of light, the time it takes for the chirp to go from minimum frequency to maximum frequency, and B is the bandwidth.

[0099] In order to determine the angle of the target relative to the radar, at least two antennas are required for reception. Figure 1 A schematic diagram shows the angle of the target relative to the FMCW radar. The first antenna measures the distance R; assuming the target is far enough away, the second antenna can be assumed to measure the distance R' = R + psinθ, where p is the distance between the two receiving antennas and θ is the angle of the target relative to the radar, which is the angle to be determined.

[0100] The expression for θ is:

[0101]

[0102] Once R and θ are determined, we can convert from polar coordinates to Cartesian coordinates:

[0103]

[0104] S2: Divide the obtained heat map dataset into training set and test set in proportion;

[0105] S3: Reference Figure 5 , improve and optimize the YOLOv8 network, including the optimization of the backbone network part, the optimization of the feature fusion part and the optimization of upsampling.

[0106] 1. Optimization of the backbone network includes:

[0107] In YOLOv8, the Spatial Pyramid Pooling-Fast (SPPF) at the end of the backbone network is a serially connected spatial pyramid pooling module used to enhance feature extraction. However, to further optimize the backbone network, this example introduces SimSPPF (Simplified Spatial Pyramid Pooling-Fast), a simplified version of the SPPF spatial pyramid pooling module.

[0108] In YOLOv8, the SimSPPF module is introduced to replace the SPPF module, refer to Figure 2The SimSPPF module includes a series of convolution operations and multiple maximum pooling and splicing operations. The convolution operation includes an initial SimConv convolution layer for preliminary processing of the input feature map, reducing the number of channels to half of the original. SimConv is a custom convolution module, including convolution operations, batch normalization, and ReLU activation function. Its function is to perform convolution operations on the input feature map to extract features, and accelerate the training process and improve the stability of the model through batch normalization; the ReLU activation function introduces nonlinearity to enhance the expressiveness of the model; the module downsamples the feature map through multiple maximum pooling operations to capture feature information at different scales. The feature map after each maximum pooling is spliced with the original feature map to achieve multi-scale feature fusion; finally, a SimConv convolution layer converts the fused feature map into a specified number of output channels to meet the requirements of subsequent network layers. Through this series of optimized designs, the SimSPPF module significantly improves the richness and accuracy of feature extraction while maintaining efficient computing performance, providing strong support for the overall performance improvement of the YOLOv8 model.

[0109] By optimizing the backbone network, the model can better extract radar heat map features.

[0110] 2. Optimization of feature fusion includes:

[0111] Bidirectional Feature Pyramid Network (BiFPN) is used for multi-scale feature fusion. BiFPN achieves more comprehensive information transmission and fusion through top-down and bottom-up bidirectional paths.

[0112] The network structure of BiFPN is as follows Figure 3 As shown in Figure 2, the core innovation of BiFPN lies in its unique bidirectional cross-scale connection and weighted feature fusion mechanism, specifically:

[0113] BiFPN simplifies the network structure and enhances information flow by optimizing cross-scale connections, removing nodes with only one input, and adding additional connections between input and output nodes at the same level. In addition, BiFPN treats each bidirectional path as a feature network layer and repeats the layer multiple times to achieve higher-level feature fusion.

[0114] In the feature fusion process, BiFPN introduces learnable weights, allowing the model to adaptively adjust the fusion method according to the importance of the features. Specifically, BiFPN uses a fast normalized fusion method to optimize the fusion results by assigning learnable weights to each input feature and dynamically adjusting these weights during training. This method not only avoids the distortion of feature information in traditional fusion methods, but also improves the model's adaptability to features of different scales. The top-down path formula in BiFPN is:

[0115]

[0116] in, is the intermediate feature of the lth layer in the top-down path, is the input feature of the lth layer, Resize is the upsampling or downsampling operation for resolution matching, Conv is the convolution operation, w1 and w2 are learnable weights, and ∈ is a small constant for numerical stability.

[0117] The bottom-up path in BiFPN is expressed as:

[0118]

[0119] in, is the output feature of the lth layer in the bottom-up path, is the intermediate feature of the top-down path, Resize is the operation for resolution matching, and w1′, w2′, and w3′ are learnable weights.

[0120] BiFPN uses neural network architecture search technology to find the optimal feature network topology; this design enables BiFPN to better integrate multi-scale features while maintaining efficient computing performance, significantly improving the model's feature expression ability and detection accuracy.

[0121] 3. Optimization for upsampling includes:

[0122] Dynamic upsampling is introduced to enhance the model's ability to recover feature maps. The dynamic upsampling process includes: first, the technology reads the original data, analyzes the data distribution, selects the basic sampling method according to the task requirements, and initializes the parameters; then, the initial sampling strategy is used to generate training batches, which are input into the model for training, and the model performance (such as loss function, accuracy, recall rate, etc.) or sample difficulty is tracked in real time; the priority or weight of the samples is dynamically calculated based on the monitoring results, such as sampling minority class samples, and the sampling probability or weight is adjusted in real time to generate new training rounds; finally, the training, monitoring, and adjustment of the sampling process are repeated to gradually optimize the model.

[0123] Dynamic upsampling specifically introduces Dysample to enhance feature recovery capabilities, designs upsampling strategies from the perspective of dynamic point sampling, and reorganizes feature maps by dynamically generating sampling points. Its structure is as follows Figure 4 Specifically, Dysample transforms the low-resolution feature map χ∈R H×W×C As input, the dynamic range adjustment factor δ is first generated by the linear transformation layer Linear1, and the value range is limited to [0, 0.5] to control the sampling range; then, another linear transformation layer Linear2 is used to generate the initial offset By combining the dynamic adjustment factor δ with the initial offset O init , generating dynamic offset O dya , the specific expression is:

[0124] O dya =O init ·δ

[0125] Then, the dynamic offset O dya Added to the fixed grid position G, a dynamic sampler S∈R is generated SH×SW×2 , define the new two-dimensional coordinates of all sampling points:

[0126] S=G+O dya

[0127] The dynamic sampler can adaptively adjust the distribution of sampling points to more accurately cover the semantic features of the target area. Based on the sampler S, the input feature map χ is reorganized by grid sampling to generate a high-resolution feature map χ′∈R SH×SW×C :

[0128] χ′=gridsample(χ, S)

[0129] The dynamic sampling mechanism makes the feature map reconstruction process more efficient by dynamically adjusting the position of the sampling points. In addition, Dysample does not rely on the generation of dynamic convolution kernels, but directly completes feature reconstruction based on the point sampling strategy, further reducing computational redundancy. Its complexity is expressed as:

[0130] T dysample =H·W·S 2 ·C

[0131] Among them, H and W represent the height and width of the input feature map respectively; S is the upsampling ratio; C is the number of channels.

[0132] S4: Train and test the optimized YOLOv8 network using the training set and test set respectively, and use the trained YOLOv8 network as the target detection model;

[0133] S5: Obtain multi-target detection results through the target detection model.

[0134] Based on the above method, the technical points of the method of the present invention can be summarized as follows:

[0135] YOLOv8's network architecture consists of three main parts: the backbone network, the neck network, and the detection head. The backbone network is responsible for extracting multi-scale features from the input image. The core modules include the C2f module, the convolution module, and the pyramid pooling module.

[0136] In the backbone network, YOLOv8 replaces the C3 module from YOLOv5, using a more efficient BottleneckBlock and residual connection design to reduce parameter count and enhance feature extraction capabilities. Its architecture includes a Split operation and stacking of multiple Bottleneck blocks, supporting dynamic gradient flow. The convolutional module, consisting of Conv2d, BatchNorm2d, and SiLU activation functions, supports downsampling and nonlinear feature mapping. Pyramid pooling, through cascaded max-pooling layers, achieves multi-scale feature fusion, significantly improving the receptive field and reducing computational redundancy.

[0137] Upsampling is used to gradually restore feature resolution, so Dysample is introduced to enhance feature recovery capabilities. The feature fusion part is responsible for fusing features at different levels and optimizing multi-scale detection capabilities. Early detectors usually make predictions directly based on the pyramid feature hierarchy extracted from the backbone network. Feature Pyramid Network (FPN) proposes a top-down approach to combine multi-scale features. Based on FPN, PANet adds an additional bottom-up path aggregation network, and then BiFPN introduces learnable weights to learn the importance of different input features, while repeatedly applying top-down and bottom-up multi-scale feature fusion to enhance the combination of semantic information and positioning information.

[0138] The head (detection head) completes target classification and positioning detection. The decoupling head separates the classification and regression tasks to reduce task interference and improve accuracy. The Anchor-Free mechanism is used to directly predict the target center point, simplify the training process, avoid anchor box hyperparameter tuning, and use Task-AlignedAssigner to dynamically allocate positive and negative samples. Combined with Distribution FocalLoss and CIoU Loss, it optimizes classification and positioning accuracy.

[0139] During model training, the required millimeter-wave radar heat map dataset is first prepared. The model size is selected according to the task. The configuration file is used to define the network structure of the backbone network, neck network, and detection head. The most appropriate loss function and training strategy are selected, and the effectiveness of the algorithm is verified through model training. The main detection process is roughly as follows: First, the image size is adjusted to an appropriate size (such as 640*640), normalized, and the channel order is converted; second, forward propagation is performed, the input image is extracted through the Backbone part, and the Neck part fuses multi-scale information. The detection head outputs the category, confidence, and coordinates of the predicted box; next, post-processing is performed to filter low-confidence prediction boxes, remove overlapping boxes through non-maximum suppression, and retain the optimal detection result; finally, the detection box is superimposed on the original image, and the category label, coordinates, and confidence are output.

[0140] Example 2:

[0141] Based on the detection method of Example 1, this embodiment provides a millimeter-wave radar multi-target detection system based on deep learning, including a peripheral interface, a memory, and a processor; the peripheral interface is used to realize data input and output during data interaction with various external devices, thereby ensuring effective connection and communication between the system and the external devices; the memory is used to store computer program instructions that can be run on the processor; the processor is used to execute the steps of a millimeter-wave radar multi-target detection method based on deep learning when running the computer program instructions.

[0142] This embodiment also provides a computer storage medium that stores a computer program that can implement the method described above when a processor executes the computer program. The computer-readable medium can be considered to be tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media include non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital tapes or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs). The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0143] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] Example 3:

[0146] In order to verify the effectiveness and effect of the method of the present invention, the following experiments were conducted in this embodiment for verification, as follows:

[0147] 1. Experimental equipment and parameters

[0148] The hardware configuration used in the experiment is as follows: the operating system is 64-bit Windows 11, the Python version is 3.8.0, the programming software used is Pycharm, the GPU model is NVIDIA GeForce GTX 1650, and the CPU model is Intel(R) Core(TM) i5-9300H CPU@2.4GHz.

[0149] 2. Dataset Collection

[0150] This experiment uses five target categories. The dataset is collected using the second-generation single-chip millimeter-wave radar AWR2944, extracting four frames of radar data per second. Each frame corresponds to a camera data frame. By converting the radar data into a Cartesian representation and recording it as a heat map, comparing the heat map with the camera data and assigning corresponding labels, the heat map can be used to detect targets such as people, bicycles, and cars. The target areas are annotated using the Roboflow tool to ensure compatibility with the YOLO model.

[0151] 3. Data Processing

[0152] The acquisition and annotation of millimeter-wave radar heatmaps is complex and time-consuming, and faces the challenge of insufficient data. To overcome this challenge and improve the generalization and robustness of the model, data augmentation techniques were used. The dataset was enhanced using the Roboflow platform. This included adding random noise ranging from -5% to +5% to simulate the noise interference that sensors might encounter in real-world applications, and adjusting image brightness. These enhancements were designed to simulate various environmental conditions likely to be encountered in real-world applications, thereby expanding the dataset and improving the model's adaptability.

[0153] 4. Model Training and Testing

[0154] During model training, Adam was used as the optimization method, with an initial learning rate of 0.01, a weight decay coefficient of 0.0005, a training batch size of 16, 100 training epochs, and an input image size of 640*640. The improved model achieved multi-object detection accuracy of 92.3% for mAP@0.5, 54.5% for mAP@0.5:0.95, 86.6% for Precision, and 89.1% for Recall, representing improvements of 2.5%, 0.3%, 1.7%, and 6.2%, respectively, compared to the baseline model. Figure 6 This is a comparison chart of the model mAP@0.5 before and after improvement. Figure 7 The comparison chart of the Precision of the models before and after improvement shows that in the early stage of training, the performance of both models improved rapidly, but the improved model showed slightly higher performance than the original model in the early stage, and in the mid-term stage, the improved model always maintained a certain advantage. In the late stage, the performance of both models tended to be stable, but the improved model was able to converge better in long-term training and achieve higher detection accuracy. In addition, the fluctuation amplitude of the improved model was relatively small, indicating that it has better stability. The confusion matrix is as follows Figure 8 As shown in the figure, the model performs well in detecting people, bicycles, and walls, but performs poorly in detecting cars and motorcycles. Further analysis and optimization of the dataset model is needed.

Claims

1. A millimeter wave radar multi-target detection method based on deep learning, characterized in that: The steps include: S1: Use millimeter-wave radar to collect echo data of multiple targets in a complex environment, process the echo data to obtain a range-angle heat map, and obtain a heat map dataset; S2: Divide the obtained heat map dataset into training set and test set in proportion; S3: Improve and optimize the YOLOv8 network; S4: Train and test the optimized YOLOv8 network using the training set and test set respectively, and use the trained YOLOv8 network as the target detection model; S5: Obtain multi-target detection results through the target detection model.

2. The millimeter wave radar multi-target detection method based on deep learning according to claim 1, characterized in that: The millimeter wave radar transmission signal in step S1 is expressed as: Where ω1(t)=2πf1(t) and f1(t) is the transmission frequency, A1 is the amplitude of the transmission signal, is the phase of the transmitted signal; The signal received by the millimeter wave radar is shown as follows: Where ω2(t)=2πf2(t) and f2(t) is the receiving frequency, A2 is the amplitude of the received signal, is the phase of the received signal; The intermediate frequency signal is obtained by mixing the received signal with the transmitted signal. The intermediate frequency signal is expressed as: The signal is low-pass filtered and expressed as follows: Apply FFT on top of the low-pass filtering to determine the frequency changes: Once the frequency change is determined, the distance is calculated using the following equation: Where c is the speed of light, the time it takes for the chirp to go from minimum frequency to maximum frequency, and B is the bandwidth.

3. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 2, wherein: In step S1, the millimeter wave radar includes two antennas, the first antenna measures the distance R; the second antenna measures the distance R'=R+psinθ, where p is the distance between the two receiving antennas and θ is the angle of the target relative to the radar; The expression for θ is: Having determined R and θ, convert from polar coordinates to Cartesian coordinates:

4. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 3, wherein: The improvement and optimization of the YOLOv8 network in step S3 includes optimization of the backbone network part, optimization of the feature fusion part and optimization of upsampling.

5. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 4, wherein: The optimization of the backbone network in step S3 includes: The SimSPPF module is introduced to replace the SPPF module. The SimSPPF module includes a series of convolution operations and multiple maximum pooling and splicing operations. The convolution operation includes an initial SimConv convolution layer for preliminary processing of the input feature map, reducing the number of channels to half of the original. SimConv is a custom convolution module, including convolution operations, batch normalization, and ReLU activation function; the ReLU activation function introduces nonlinearity to enhance the expressiveness of the model; the module downsamples the feature map through multiple maximum pooling operations to capture feature information of different scales. The feature map after each maximum pooling will be spliced with the original feature map to achieve multi-scale feature fusion; finally, a SimConv convolution layer is used to convert the fused feature map into the specified number of output channels.

6. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 4, wherein: The optimization of the feature fusion part in step S3 includes: BiFPN is used for multi-scale feature fusion. BiFPN achieves more comprehensive information transmission and fusion through top-down and bottom-up bidirectional paths.

7. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 6, wherein: The BiFPN in step S3 includes a bidirectional cross-scale connection and a weighted feature fusion mechanism, specifically: BiFPN simplifies the network structure and enhances information flow by optimizing cross-scale connections, removing nodes with only one input, and adding additional connections between input and output nodes at the same level; In addition, BiFPN treats each bidirectional path as a feature network layer and repeats the layer multiple times to achieve higher-level feature fusion; In the feature fusion process, BiFPN introduces learnable weights, allowing the model to adaptively adjust the fusion method according to the importance of the features; Specifically, BiFPN adopts a fast normalized fusion method to optimize the fusion results by assigning learnable weights to each input feature and dynamically adjusting these weights during training; BiFPN uses neural network architecture search technology to find the optimal feature network topology; The top-down path formula in BiFPN is: in, is the intermediate feature of the lth layer in the top-down path, is the input feature of the lth layer, Resize is the upsampling or downsampling operation for resolution matching, Conv is the convolution operation, w1 and w2 are learnable weights, ∈ is a small constant for numerical stability; The bottom-up path in BiFPN is expressed as: in, is the output feature of the lth layer in the bottom-up path, is the intermediate feature of the top-down path, Resize is the operation for resolution matching, and w1′, w2′, and w3′ are learnable weights.

8. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 4, wherein: The optimization of upsampling in step S3 includes: Introducing dynamic upsampling to enhance the model's ability to recover feature maps; Dysample is introduced to enhance the feature recovery capability, and an upsampling strategy is designed from the perspective of dynamic point sampling. The feature map is reorganized by dynamically generating sampling points. Specifically, Dysample converts the low-resolution feature map χ∈R H×W×C As input, the dynamic range adjustment factor δ is first generated by the linear transformation layer Linear1, and the value range is limited to [0, 0.5] to control the sampling range; then, another linear transformation layer Linear2 is used to generate the initial offset By combining the dynamic adjustment factor δ with the initial offset O init , generating dynamic offset O dya , the specific expression is: The dya =O init ·d Then, the dynamic offset O dya Added to the fixed grid position G, a dynamic sampler S∈R is generated SH×SW×2 , define the new two-dimensional coordinates of all sampling points: S=G+O dya The dynamic sampler can adaptively adjust the distribution of sampling points to more accurately cover the semantic features of the target area. Based on the sampler S, the input feature map χ is reorganized by grid sampling to generate a high-resolution feature map χ′∈R SH×SW×C : χ′=gridsample(χ, S) The dynamic sampling mechanism dynamically adjusts the position of the sampling points, and its complexity is expressed as: T dysample =H·W·S 2 ·C Among them, H and W represent the height and width of the input feature map respectively; S is the upsampling ratio; C is the number of channels.

9. The method for multi-target detection using millimeter-wave radar based on deep learning according to claim 8, wherein: The dynamic upsampling process in step S3 includes: reading the original data, analyzing the data distribution, selecting a basic sampling method according to the task requirements, and initializing the parameters; then, using the initial sampling strategy to generate a training batch, inputting it into the model for training, and tracking the model performance or sample difficulty in real time; dynamically calculating the priority or weight of the sample based on the monitoring results, and adjusting the sampling probability or weight in real time to generate a new training round; finally, repeating the training, monitoring, and adjusting the sampling process to gradually optimize the model.

10. A millimeter-wave radar multi-target detection system based on deep learning, characterized by: Includes peripheral interfaces, memory and processor; The peripheral interface is used to implement data input and output during data interaction with various external devices, thereby ensuring effective connection and communication between the system and the external devices; The memory is configured to store computer program instructions that can be executed on the processor; The processor is configured to execute the steps of a millimeter-wave radar multi-target detection method based on deep learning according to any one of claims 1 to 9 when running the computer program instructions.

Images