Millimeter wave image detection method and system based on heterogeneous MPSoC

Abstract

The invention discloses a millimeter wave image detection method and system based on a heterogeneous MPSoC, and the method comprises the steps: taking YOLOv5s as a baseline model for improvement at an algorithm level, reconstructing a backbone network based on a C3-Ghost module, introducing a channel-space dual attention mechanism, and enhancing the sensitivity to a weak reflection target through feature grouping reconstruction and dynamic weight distribution. And finally, an improved Soft-NMS algorithm based on Gaussian attenuation is adopted to optimize a redundant frame suppression strategy in a dense scene. On the hardware acceleration level, based on a ZynqMP heterogeneous platform and a Viis AI tool chain, a'preprocessing-reasoning-scheduling 'three-level acceleration architecture is constructed. The millimeter wave image detection precision is high, the problems of false detection and missing detection are reduced, the real-time performance is good, and the power consumption is low.

Description

Technical Field

[0001] The present invention relates to the field of millimeter wave imaging technology, and in particular to a millimeter wave image detection method and system based on heterogeneous MPSoC. Background Art

[0002] Millimeter-wave imaging technology, with its non-contact penetration and high resolution, has demonstrated significant value in public security, medical diagnostics, and industrial inspection. Compared to traditional X-ray and visible light imaging, millimeter-wave (30-300 GHz) can penetrate non-metallic materials such as clothing and plastic, accurately capturing the reflective characteristics of hidden objects on the human body or within objects, while avoiding the risk of ionizing radiation. These characteristics make it a core technology for detecting hidden objects on the human body in scenarios such as airport security and counter-terrorism screening. However, the technology still faces numerous challenges in its engineering application. For one thing, millimeter-wave images suffer from low signal-to-noise ratios and blurred object edges, making it difficult for traditional millimeter-wave hidden object detection algorithms based on handcrafted features to achieve the accuracy required for practical applications. Furthermore, while deep learning technologies, such as convolutional neural networks (CNNs), have significantly improved the performance of hidden object detection models in security inspection scenarios through end-to-end feature learning, the resulting surge in model size and computational complexity has placed stricter demands on the real-time processing capabilities and energy efficiency of hardware platforms.

[0003] Current mainstream object detection models (such as the YOLO series and Faster R-CNN) face two major challenges in millimeter-wave imagery scenarios. First, millimeter-wave images have limited resolution and weak target features, requiring deep networks to extract multi-scale semantic information. This also leads to a sharp increase in the number of model parameters and computational complexity. Second, security inspection scenarios have strict requirements for real-time performance and energy consumption, and traditional CPU / GPU-based solutions struggle to balance high throughput and low power consumption. For example, while GPUs offer high computing power, their high power consumption makes them difficult to adapt to portable security inspection equipment. The serial architecture of CPUs cannot meet the parallel computing requirements of deep learning models, resulting in excessively high inference latency. Against this backdrop, FPGA (field-programmable gate array)-based hardware acceleration technology has become a key approach to addressing these issues.

[0004] FPGAs, with their reconfigurability, parallel computing capabilities, and high energy efficiency, provide a new solution to the contradiction between real-time constraints and computational complexity faced by millimeter-wave image hidden object detection algorithms. First, FPGAs support customized data path design and can build dedicated hardware modules for core operations such as convolution operations and feature fusion, significantly improving computational efficiency. Second, FPGAs' pipeline architecture and on-chip storage optimization strategies can effectively alleviate memory bandwidth bottlenecks, reduce data handling overhead, and thus shorten processing delays. In addition, the dynamic reconfiguration characteristics of FPGAs enable them to flexibly adapt to algorithm iteration requirements, support rapid adjustment of hardware architecture for different security inspection scenarios (such as changes in human posture and multi-target detection), and reduce system upgrade costs.

[0005] (1) Millimeter wave detection technology

[0006] In the early days of hidden object detection in millimeter-wave imagery, automatic threshold segmentation methods based on histogram analysis (such as the Otsu algorithm) were widely used due to their simplicity and efficiency. Lee et al. achieved effective segmentation of concealed weapons in millimeter-wave imagery by improving the threshold weighting strategy. However, their performance was highly dependent on the bimodal distribution of the image and was sensitive to noise in low-contrast images. In 2008, Shen et al. proposed a method combining anisotropic diffusion denoising and multi-level threshold segmentation. This method used a Gaussian mixture model to model the image intensity distribution and used contour evolution to locate targets, significantly improving the accuracy of hidden object detection. However, their proposed algorithm was complex and difficult to meet real-time requirements.

[0007] To address multi-target detection in complex backgrounds, object detection methods based on statistical models and traditional machine learning have gradually emerged. Du Kun et al. established a statistical model of human regions based on the generalized composite distribution (GC-PDF) and the generalized K distribution (GK-PDF). Combined with constant false alarm rate (CFAR) detection technology, they achieved a detection rate of 93.51% in active millimeter-wave imagery. However, their computational process relies on complex parameter estimation and has limited adaptability to non-uniform backgrounds. Yeom et al. proposed a multi-level expectation maximization (EM) algorithm, which uses a two-level Gaussian mixture model to segment human regions and hidden objects, respectively. The average error rate is reduced by 58.7% compared to traditional EM. Zheng et al. further combined SIFT, HOG, and saliency features, using sparse coding and a linear SVM classifier to achieve near-real-time detection (600ms / frame) in synthetic aperture radar (SAR) imagery. They also improved generalization through data augmentation. In summary, while traditional hidden object detection algorithms in millimeter-wave imagery can achieve good detection results under certain conditions, they also have significant limitations. Most of these algorithms rely on simple statistical features of images or fixed model assumptions, making them difficult to adapt to complex and changing real-world scenarios. They also place high demands on image quality, and their detection performance degrades dramatically when faced with low contrast, strong noise interference, or when the target and background characteristics do not meet the preset requirements.

[0008] With the development of deep learning technology, object detection methods based on convolutional neural networks (CNNs) have gradually become mainstream. For example, Liu et al. introduced Faster R-CNN to millimeter-wave image detection. They enhanced feature resolution through dilated convolution and improved small object detection performance by combining a contextual embedding module. The large-scale dataset they constructed (58k images) significantly improved the model's generalization ability. However, two-stage detectors like Faster R-CNN suffer from high computational complexity due to region generation and secondary classification, making them difficult to meet real-time requirements. To address this issue, Hou Bingji et al. improved Faster R-CNN by recovering low-order features through deconvolution and introducing the Online Hard Example Mining (OHEM) technique to optimize sample balance. This method ultimately increased detection speed to 27 frames per second.

[0009] To balance real-time performance and accuracy, single-stage detectors (such as YOLO) have gradually attracted attention. WANG et al. introduced the Normalized Accumulation Map (NAM) in YOLO-v2, leveraging spatial prior information about object locations in millimeter-wave images to dynamically adjust the confidence loss weights. This improved mean average precision (mAP) by 4.43% while maintaining high efficiency. YANG et al. combined human keypoint detection with a deformable body partitioning (DBP) model, using YOLOv3 to dynamically identify hidden objects within each body partition, further optimizing detection stability. FAN et al. proposed an enhancement method based on conditional generative adversarial networks (CGANs). This method alleviates the problem of small sample training by generating high-fidelity millimeter-wave images. Their effectiveness was verified by combining it with YOLOv4, achieving an improvement of approximately 2% in mean average precision (mAP) above a confidence threshold of 0.5, with significant feature enhancement in low-contrast scenes. In summary, the YOLO series currently has significant advantages in speed and resource consumption due to its single-stage end-to-end architecture. Therefore, combining YOLO's fast inference capabilities with targeted optimization strategies (such as lightweighting or feature enhancement) is expected to achieve a balance between speed and accuracy in millimeter-wave image detection while laying the model foundation for subsequent edge device deployment.

[0010] (2) FPGA hardware acceleration technology

[0011] In recent years, artificial intelligence (AI) applications have permeated nearly every emerging technology sector. The computing power demand driven by its large-scale adoption has increased exponentially, creating a computing bottleneck that has become a fundamental obstacle to the development of AI technology. Against this backdrop, a technology approach centered around dedicated hardware accelerators has gradually become mainstream. FPGAs (field-programmable gate arrays), with their reconfigurability, hardware parallel processing capabilities, and high energy efficiency, have demonstrated unique competitiveness in deep learning acceleration.

[0012] FPGA implementations of convolutional neural networks (CNNs) date back to the mid-1990s, with early explorations centered around hardware architecture innovation. Cloutier et al. pioneered the construction of a virtual image processor (VIP) on the Altera EPF81500 FPGA. This VIP employed a SIMD multiprocessor architecture with a two-dimensional ring topology connecting processing elements (PEs). This VIP simplified multiplier design by using low-precision arithmetic to improve performance. The resulting image processor provided a model for subsequent hardware acceleration research. With the evolution of FPGA chip technology, modern devices have integrated a large number of MAC units optimized for digital signal processing, making CNN implementations on FPGAs extremely energy-efficient.

[0013] Against this backdrop, researchers have gradually shifted their focus to optimizing convolutional layers to accelerate computing engines. Farabet et al. developed a convolutional network processor (CNP). By leveraging hardware-dedicated convolution modules and 16-bit fixed-point quantization, the CNP achieves real-time image processing while reducing resource usage, successfully demonstrating the feasibility of deploying deep networks on FPGAs. Zhang et al. introduced the roofline model and, using loop unrolling and blocking strategies, systematically optimized the parallel computing and memory access capabilities of the convolutional layer. Their resulting design, an accelerator on the VC707 board, achieves a throughput of 61.62 GOP / s.

[0014] At the same time, scalable accelerator architectures have gradually become a research hotspot. SUDA et al. designed a dynamically configurable accelerator based on the OpenCL framework. Using genetic algorithms to automatically optimize hardware parameters, they achieved a VGG-16 inference throughput of 117.8 GOP / s on a Stratix-V FPGA. VENIERI et al. proposed an FPGA-based ConvNet framework based on Synchronous Data Flow (SDF). This framework supports dynamic subgraph partitioning and weight reloading, achieving low-latency inference on the Zynq platform while significantly improving the flexibility of complex model deployment.

[0015] To overcome computing bottlenecks, researchers are exploring the potential of parallelization from multiple dimensions. Liu et al. designed a four-level parallel framework: task-level, layer-level, loop-level, and operation-level, achieving a VGG-S network inference throughput of 473.4 GOP / s on a Virtex7 FPGA. Umuroglu et al., taking a different approach, using a binarized neural network (BNN), the Winograd algorithm, and approximate computing techniques, pushed CNV network throughput to 2,465.5 GOP / s, pioneering ultra-low bit quantization technology.

[0016] In summary, existing millimeter wave imaging technology has the following defects:

[0017] (1) Low accuracy

[0018] Millimeter wave images have problems such as low signal-to-noise ratio and blurred target edges, which makes it difficult for traditional millimeter wave hidden object detection algorithms based on manual features to achieve the accuracy requirements required for practical applications.

[0019] (2) Poor real-time performance, large hardware resource usage, and high power consumption

[0020] Security inspection scenarios have stringent requirements for real-time performance and energy efficiency. Traditional CPU / GPU-based solutions struggle to balance high throughput with low power consumption. For example, while GPUs offer high computing power, their high power consumption makes them difficult to adapt to portable security inspection equipment. Meanwhile, the serial architecture of CPUs cannot meet the parallel computing requirements of deep learning models, resulting in excessively high inference latency. Summary of the Invention

[0021] To address the problems of low precision, poor real-time performance, high hardware resource usage, and high power consumption in existing target detection models in millimeter-wave image scenarios, this paper proposes an FPGA-based hardware accelerator design. By combining algorithm lightweighting with hardware architecture optimization, this approach achieves high-precision, low-power, real-time detection. At the algorithmic level, improvements are made using YOLOv5s as the baseline model. The backbone network is reconstructed based on the C3-Ghost module, and a channel-spatial dual attention mechanism is introduced. Sensitivity to weakly reflective targets is enhanced through feature grouping reconstruction and dynamic weight allocation. Finally, an improved Soft-NMS algorithm based on Gaussian attenuation is used to optimize the redundant box suppression strategy in dense scenes. At the hardware acceleration level, a three-level acceleration architecture of "preprocessing-inference-scheduling" is constructed based on the ZynqMP heterogeneous platform and the Vitis AI toolchain.

[0022] In view of this, the technical solution adopted by the present invention is a millimeter wave image detection method based on heterogeneous MPSoC, comprising the following steps:

[0023] The backbone network uses the YOLOv5 algorithm, including the Backbone layer, Neck layer, head layer, and post-processing stage;

[0024] The Backbone layer preprocesses the millimeter wave image, and the C3-Ghost module is used to replace the original C3 module in the Backbone layer;

[0025] The Neck layer performs hidden object detection on the preprocessed image, and then enters the SA module of the channel-spatial dual attention mechanism to perform feature channel rearrangement and channel and spatial attention calculation;

[0026] The head layer performs convolution on the output of the Neck layer and then enters the post-processing stage;

[0027] In the post-processing stage, the Soft-NMS algorithm is used to replace the traditional NMS, and the Gaussian attenuation function is used to optimize the frame suppression strategy between dense targets;

[0028] Output millimeter wave hidden object detection results.

[0029] Furthermore, the C3-Ghost module adopts a Ghost bottleneck layer, and the Ghost bottleneck layer adopts a dual-module cascade architecture: the first-layer Ghost module increases the channel dimension through a feature expansion mechanism, and the second-layer Ghost module implements channel compression to maintain the consistency of input and output dimensions; feature fusion is achieved between modules through residual connections, and the back-end of each sub-module implements batch normalization and ReLU activation function processing in turn.

[0030] Furthermore, a depthwise separable convolutional layer is embedded between the two Ghost modules.

[0031] Furthermore, the SA module includes the following steps:

[0032] ① Feature grouping

[0033] For a given feature Figure X , first divide it into G groups of sub-features along the channel dimension

[0034] X=[X1,X2,...,X G ],X k ∈R C / G×H×W (1.1)

[0035] Each group of sub-features X k Further split into two branches X k1 With X k2 , processing channel attention and spatial attention respectively; C represents the number of channels, for example, a color image has three RGB channels c = 3; R C / G×H×W Represents a 4D real tensor.

[0036] ② Channel attention

[0037] Channel attention generates channel statistics through global average pooling, and then uses a simple gating mechanism with sigmoid activation to generate the final channel attention output. The calculation formulas are shown in (1.2) and (1.3):

[0038]

[0039] X' k1 =σ(f c (s))·X' k1 =σ(W1s+b1)·X k1 (1.3)

[0040] Where W1 and b1 are parameters for scaling and offset, s represents the generated channel statistics, and X' k1 Represents the channel attention output information, H represents the height, W represents the width, i and j represent the index of the feature map in the spatial dimension, f gprepresents the global pooling operation, σ represents the activation function, f c (s) represents a fully connected layer, which is used to perform feature transformation on the vector s after global pooling.

[0041] ③Spatial attention

[0042] Spatial attention generates spatial statistics through group normalization, and then uses a simple gating mechanism with sigmoid activation to generate the final spatial attention output, which is calculated as shown in formula (1.4)

[0043] X' k2 =σ(W2·GN(X k2 )+b2)·X k2 (1.4)

[0044] Where W2 and b2 are parameters, X' k2 Represents spatial attention output information;

[0045] ④ Aggregation and channel reorganization

[0046] The outputs of the two branches are concatenated along the channel dimension to form X' k =[X' k1 ,X' k2 ], then aggregate all sub-features and promote information interaction between groups through channel shuffling operation, and finally output a feature map with the same size as the input.

[0047] Furthermore, the calculation of the Soft-NMS algorithm is shown in formula (1.7):

[0048]

[0049] B max represents the detection box with the highest confidence, θ represents the IoU threshold, IOU represents the intersection-over-union ratio between the detection box and the reference box, S i Represents the detection box B j Confidence score of .

[0050] The present invention also provides a millimeter wave image detection system based on heterogeneous MPSoC, which is collaboratively composed of a processor system (PS) and a programmable logic terminal (PL); the Backbone layer, Neck layer and head layer in the above detection method are mapped to the programmable logic terminal (PL), and the post-processing stage is mapped to the processor system (PS);

[0051] The processor system (PS) includes an ARM, which reads the millimeter wave image and enables DMA and Letterbox IP cores to pre-process the image;

[0052] The programmable logic (PL) includes a Letterbox IP core and a DPU IP core. The Letterbox IP core pre-processes the image, and the pre-processed image is sent to the DPU IP core for hidden object detection.

[0053] The ARM collects the data transmitted by the DPU IP core and decodes and performs post-processing operations on it, and finally visualizes the final millimeter wave hidden object detection results on the ARM side.

[0054] Furthermore, a heterogeneous communication channel is established between the PS and PL via an AXI high-speed bus, and a DMA transmission mechanism is used to achieve efficient interaction between detection data and image data.

[0055] Furthermore, the processor system (PS) also performs data scheduling, parameter configuration of the hardware acceleration IP core, and calling of the underlying driver library; the ARM builds a data processing pipeline based on the Python language under the Linux operating system environment, and implements the analysis and optimization of the prediction results by calling Numpy and OpenCV, and finally completes the visualization of the target detection results.

[0056] Furthermore, the programmable logic terminal (PL) further includes a ZYNQMP processor, a clock frequency division module, an AXI-DMA controller and an AXI HP interface;

[0057] The clock divider module divides the 100MHz reference clock provided by the ZynqMP processor into multiple signals. The 325MHz clock simultaneously drives the convolution calculation array of the DPU IP core, the Letterbox IP core pre-processing pipeline, the AXI-DMA controller, and the AXI HP interface; the 650MHz clock is used for the calculation unit of the DPU IP core; and the 100MHz clock is used for the AXI-Lite configuration bus, reset logic, and peripheral control module.

[0058] The AXI-DMA controller is used for high-speed transmission of image preprocessing and inference results. Its MM2S channel reads raw data from DDR at a 325MHz clock and pushes it to the Letterbox IP core buffer via the AXI4-Stream interface. The S2MM channel receives the preprocessing results output by the Letterbox IP core and writes them back to DDR for processing and analysis by the DPU.

[0059] The ZYNQMP processor, serving as the system control hub, coordinates hardware resources and task scheduling through the Linux driver layer. The software side calls Vitis AI Runtime to load the quantized MSP-YOLOv5 model, dynamically configures the input and output size parameters of the Letterbox IP core, and uses the interrupt controller to respond to the DPU IP core's inference completion events and DMA transfer status, enabling pipelined coordinated scheduling of pre-processing, inference, and post-processing tasks.

[0060] The AXI interconnect uses a hierarchical crossbar switch architecture to connect the DPU, AXI-DMA controller, DDR controller, and peripheral modules.

[0061] Furthermore, the processor system (PS) also includes a DDR controller, and the DDR storage space is divided into an input image buffer area, a preprocessing result area and a model weight area.

[0062] The present invention has the following beneficial technical effects:

[0063] (1) Reduce model complexity and improve model detection accuracy

[0064] By introducing the C3-Ghost module to reconstruct the backbone network, the cross-scale feature representation capability of blurred targets is enhanced; using the channel-space dual attention mechanism (Shuffle Attention) to group feature maps and dynamically assign weights, the sensitivity to weakly reflective targets is improved, thereby improving the accuracy of hidden object detection while reducing the complexity of the model.

[0065] (2) Reduce false detection and missed detection problems

[0066] An improved Soft-NMS algorithm based on Gaussian attenuation is adopted to optimize the redundant box suppression strategy in dense scenes, effectively alleviating the problems of false detection and missed detection of hidden objects in dense scenes.

[0067] (3) Good real-time performance

[0068] Based on the ZynqMP heterogeneous platform and the Vitis AI tool chain, a three-level acceleration architecture of "preprocessing-inference-scheduling" is constructed. Through the hardware acceleration module of the preprocessing layer, the deep learning processing unit of the inference layer, and the coordinated scheduling mechanism of software and hardware tasks at the system layer, the system processing speed is significantly improved to meet the requirements of real-time detection.

[0069] (4) Low power consumption

[0070] At the hardware acceleration level, by optimizing storage bandwidth, dynamically allocating URAM resources, and co-designing software and hardware, the entire process from model quantization and compilation to deployment is accelerated, significantly improving the system energy efficiency and enabling the entire system to maintain efficient operation while maintaining low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is the network architecture of the MSP-YOLOv5 algorithm;

[0072] Figure 2 For the Ghost module;

[0073] Figure 3 It is the Ghost bottleneck layer;

[0074] Figure 4 It is a backbone network architecture based on C3-Ghost modules;

[0075] Figure 5 It is the SA model architecture;

[0076] Figure 6 It is the hardware accelerator system architecture;

[0077] Figure 7 Accelerate system workflow for hardware;

[0078] Figure 8 A schematic diagram for building a hardware platform based on DPU and Letterbox;

[0079] Figure 9 Provide a flowchart for the overall deployment;

[0080] Figure 10 The power consumption analysis results of the accelerator. DETAILED DESCRIPTION

[0081] 1. Design of hidden object detection algorithm for millimeter wave images in low signal-to-noise ratio scenarios

[0082] (1) Improvement of the present invention on model lightweighting

[0083] The YOLOv5 network achieves good performance in object detection tasks, but its backbone network architecture suffers from high inference latency and large memory usage when running on edge devices, seriously impacting the real-time detection capabilities of millimeter-wave security inspection systems. Therefore, this paper uses the C3-Ghost lightweight network model to improve the lightweight YOLOv5 backbone network.

[0084] In order to verify the improvement effect of the lightweight model, the present invention compares various improved models based on a self-made millimeter-wave image dataset, and uses ShuffleNetV2 and MobileNetV3 models to replace the backbone network, and uses C3-Faster and C3-Ghost modules to replace the C3 module in the backbone network.

[0085] The performance of the baseline model and various improved models is shown in Table 1. As can be seen from the table, replacing the YOLOv5 backbone network with ShuffleNetV2 and MobileNetV3 achieves excellent lightweighting, significantly reducing model parameters, floating-point operations, and model size compared to the baseline model. However, while computational cost decreases, replacing the backbone network also leads to a significant decrease in detection performance (mAP). By replacing the C3 module, the performance of the YOLOv5 model is more balanced. In particular, C3-Ghost-YOLOv5 improves mAP by 2.1 percentage points, despite reducing model parameters, floating-point operations, and model size by 16.6%, 21.8%, and 16.1%, respectively. Considering the high-precision requirements of millimeter-wave security inspection systems, it is appropriate to select the C3-Ghost module with the best overall performance to replace the YOLOv5 backbone network.

[0086] Table 1 Comparison results of various model experiments

[0087]

[0088] (2) Millimeter wave image hidden object detection algorithm based on MSP-YOLOv5

[0089] In actual detection tasks, the physical characteristics of hidden targets in millimeter wave images are slightly different from the background radiation. At the same time, affected by the penetrating imaging characteristics of the imaging equipment, the edge contours and texture features of the target are relatively blurred. These millimeter wave image characteristics significantly increase the difficulty of feature extraction and target positioning. In order to enhance the model's ability to capture the features of hidden targets and adapt to the real-time detection requirements of millimeter wave security inspection scenarios, this paper improves the YOLOv5 algorithm and proposes the MSP-YOLOv5 algorithm. The network architecture of this algorithm is as follows: Figure 1 shown.

[0090] The MSP-YOLOv5 algorithm primarily makes the following improvements to the Backbone layer, Neck layer, and post-processing stages of YOLOv5: 1) The Backbone layer replaces the original C3 module with the C3-Ghost module. By introducing depthwise separable convolution and feature map reconstruction mechanisms, a lightweight feature extraction unit is constructed, enhancing the ability to extract weak reflection features while maintaining computational efficiency. 2) The Neck layer introduces the Shuffle Attention module, which uses a channel-spatial dual attention mechanism. This improves sensitivity to the subtle features of hidden objects through feature channel rearrangement and spatial attention weighting. 3) To address the problem of missed detections caused by mutual suppression of adjacent hidden target detection boxes in dense scenes, the Soft-NMS algorithm is used in the post-processing stage to replace the traditional NMS. This algorithm uses a Gaussian attenuation function to optimize the box suppression strategy between densely packed objects, effectively reducing the probability of false suppression between adjacent hidden objects.

[0091] 1) C3-Ghost

[0092] The core module of the GhostNet architecture is the Ghost module, and the module structure is as follows Figure 2 As shown in the figure, the Ghost module leverages the similarities between feature maps to generate basic feature maps through a small number of standard convolutions. These basic feature maps are then linearly combined (similar to weighted summation in mathematics) to generate new feature maps in batches. Through this parameter reuse mechanism, the neural network maintains the model's ability to recognize complex features while significantly reducing computational and storage requirements, providing an efficient implementation solution for mobile deployment.

[0093] Based on the lightweight advantage of the Ghost module, Han et al. further constructed the Ghost bottleneck layer (GhostBottleneck). The Ghost bottleneck layer is as follows Figure 3 shown.

[0094] The Ghost bottleneck layer draws on the principles of ResNet residual networks and adopts a two-module cascade architecture: the first-layer Ghost module increases channel dimensions through feature expansion, while the second-layer Ghost module implements channel compression to maintain input and output dimensionality consistency. Residual connections (shortcuts) are used between modules to achieve feature fusion, and each sub-module is sequentially processed with batch normalization (BN) and ReLU activation functions at the back end.

[0095] In terms of activation function application strategy, the Ghost bottleneck layer inherits the advantages of the linear bottleneck structure of MobileNetv2. ReLU activation is used after the first Ghost module. This has the advantage of increasing network nonlinearity while effectively avoiding information loss in the low-dimensional feature space, ensuring feature integrity and validity.

[0096] When the feature map size needs to be reduced (for example, by setting the stride to 2), the Ghost bottleneck layer innovatively embeds a depthwise separable convolution (DWConv) layer between two Ghost modules. Depthwise separable convolution is a highly efficient convolution method that decomposes the standard convolution into depthwise and pointwise convolutions, significantly reducing the number of parameters and computation. This design not only compresses the feature map size and reduces the complexity of subsequent computations, but also enhances the network's robustness to geometric distortions, enabling the network to maintain good feature extraction capabilities despite changes such as image rotation and scaling.

[0097] Similar to the C3-Faster structure, the lightweight backbone network structure is as follows Figure 4 As shown in Figure 1, the present invention replaces the residual component within the C3 module with a Ghost bottleneck layer containing a Ghost module, ultimately obtaining a reconstructed C3-Ghost module. This module can extract more feature information while operating at a low cost, thereby reducing model complexity.

[0098] 2) Neck layer improvement based on attention mechanism

[0099] In the target detection task, the Neck layer serves as an important bridge connecting the Backbone feature extraction network and the detection head, and plays a key role in multi-scale feature fusion and semantic information enhancement. However, when it comes to detecting hidden targets in millimeter-wave images, the traditional feature pyramid network is extremely susceptible to interference from background noise during cross-scale feature fusion, which will also make it difficult to identify weak feature signals. To this end, the present invention chooses to introduce an attention mechanism in the Neck layer, and enhances the network's ability to focus on key target features by establishing a dynamic feature selection mechanism across channels and spatial dimensions. Compared to the problem of over-screening of underlying features that may result from directly applying the attention module to the Backbone layer, the attention module added to the Neck layer can act more accurately on high-order features that have undergone deep abstraction. It strengthens the response intensity of hidden targets by rearranging feature channels, and uses spatial attention to suppress interference from irrelevant areas, providing a more discriminative multi-scale feature representation for subsequent detection heads.

[0100] In the field of neural networks, the attention mechanism occupies a pivotal position. It gives the neural network the ability to filter input information, enabling it to focus on elements closely related to the task, thereby significantly improving the performance of the model. In computer vision research, there are two main attention mechanisms: spatial attention and channel attention. The spatial attention mechanism focuses on capturing the correlation between different pixels in the image to determine the key areas that need attention, while the channel attention mechanism goes deep into the channel level, exploring the mutual dependencies between channels, and providing the network with more discriminative feature representations. Although combining these two attention mechanisms can achieve better performance, it will inevitably increase the computational overhead. To solve this problem, Qing-Long Zhang and Yu-Bin Yang proposed an efficient Shuffle Attention (SA) module. The core idea of ​​this module is to organically integrate the advantages of spatial attention and channel attention in an efficient way, so as to fully explore the potential of the two to work together without significantly increasing the computational burden, and ultimately achieve a leap in the performance of deep convolutional neural networks (CNNs). The model architecture is as follows. Figure 5 shown.

[0101] SA first divides the input feature map into multiple groups according to the channel dimension, then uses the Shuffle Unit for each group to capture feature dependencies in both spatial and channel dimensions. Finally, the channel shuffle operation is used to enable information exchange between different sub-features.

[0102] ① Feature grouping

[0103] For a given feature Figure X , SA first divides it into G groups of sub-features along the channel dimension

[0104] X=[X1,X2,...,X G ],X k ∈R C / G×H×W (1.1)

[0105] Each group of sub-features X k Further split into two branches X k1 With X k2 , processing channel attention and spatial attention respectively.

[0106] ② Channel attention

[0107] Channel attention generates channel statistics through global average pooling (GAP), and then uses a simple gating mechanism with sigmoid activation to generate the final channel attention output. The calculation formulas are shown in (1.2) and (1.3):

[0108]

[0109] X' k1 =σ(f c (s))·X' k1 =σ(W1s+b1)·X k1 (1.3)

[0110] Where W1 and b1 are parameters for scaling and offset, s represents the generated channel statistics, and X' k1 Represents the channel attention output information. H represents the height, W represents the width, i and j represent the index of the feature map in the spatial dimension, respectively, f gp represents the global pooling operation, σ represents the activation function, f c (s) represents a fully connected layer, which is used to perform feature transformation on the vector s after global pooling.

[0111] ③Spatial attention

[0112] Spatial attention generates spatial statistics through group normalization (GN), and then uses a simple gating mechanism with sigmoid activation to generate the final spatial attention output, which is calculated as shown in formula (1.4)

[0113] X' k2 =σ(W2·GN(X k2 )+b2)·X k2 (1.4)

[0114] Where W2 and b2 are parameters, X' k2 Represents the spatial attention output information.

[0115] ④ Aggregation and channel reorganization

[0116] The outputs of the two branches are concatenated along the channel dimension to form X' k =[X' k1 ,X' k2 ], then aggregates all sub-features and promotes information interaction between groups through a channel shuffle operation, ultimately outputting a feature map with the same size as the input. Channel shuffling breaks down group isolation and enhances feature diversity by periodically permuting the channel order.

[0117] 3) Non-maximum algorithm optimization

[0118] In target detection algorithms, non-maximum suppression (NMS) is a key post-processing step. Traditional NMS algorithms play an important role in eliminating redundant detection frames and determining the final target location and category. However, in the millimeter-wave hidden object detection scenario, traditional NMS suppresses overlapping frames through a fixed threshold strategy. When multiple hidden objects are closely adjacent, this strategy is extremely likely to lead to missed detection of dense targets. In order to improve detection accuracy and efficiency, the present invention uses the Soft-NMS algorithm to improve the non-maximum suppression algorithm to better adapt to the millimeter-wave hidden object detection task.

[0119] The main idea of ​​the traditional NMS algorithm is to sort the detection results for each category from high to low according to the confidence score of the detection box. The detection box with the highest confidence score is selected as the reference box, and then the intersection over union (IoU) of the remaining detection boxes with the reference box is calculated. The IoU is generally used to indicate the degree of overlap between the reference box and the detection box. If the IoU between a detection box and the reference box is greater than a set threshold, the detection box is considered redundant and is suppressed (usually removed from subsequent results). This process is repeated until all detection boxes are processed, ultimately obtaining a set of non-redundant detection boxes for each category.

[0120] Assume there is a set of detection boxes {B i}, and its corresponding confidence score {S i}, S i Sort from high to low. For each category, select the detection box B with the highest confidence max , for the remaining detection boxes B j , IoU calculation is shown in formula (1.5):

[0121]

[0122] Among them, Area(B max ∩B j ) is the detection box B max and B j The area of ​​the intersection, Area(B max ∪B j ) is the area of ​​their union. The non-maximum suppression algorithm is shown in formula (3-8):

[0123]

[0124] Where θ is the IoU threshold, if IOU(B max ,B j )>θ, then suppress B j .

[0125] The Soft-NMS algorithm is an improvement on the traditional NMS. Its core idea is not to directly delete the detection boxes whose IoU is greater than the threshold, but to reduce the confidence scores of these detection boxes and re-screen the detection boxes based on the adjusted confidence scores. This can retain the information of the detection boxes that may have been mistakenly suppressed to a certain extent, thereby improving the accuracy of detection. The calculation of this function is shown in formula (1.7):

[0126]

[0127] B max represents the detection box with the highest confidence, θ represents the IoU threshold, IOU represents the intersection-over-union ratio between the detection box and the reference box, S i Represents the detection box B j Confidence score of .

[0128] (3) Model comparison results

[0129] To validate the performance of the MSP-YOLOv5 model, we selected mainstream object detection methods: SSD, Faster R-CNN, and YOLOv11s, and conducted comparative experiments in a unified experimental environment. Both SSD and Faster R-CNN used ResNet50 as their backbone networks. Evaluation metrics included Params, GFLOPS, Precision, Recall, and mAP@50. The experimental results are shown in Table 2.

[0130] Table 2 Performance comparison of each model

[0131]

[0132] Table 2 shows that Faster R-CNN and SSD are less computationally expensive and less accurate than the YOLO family of algorithms. Among the YOLO family of algorithms, the lightweight model MSP-YOLOv5 proposed in this paper performs the best, achieving optimal performance in both accuracy and computational cost evaluation metrics. Specifically, compared to YOLOv11s, MSP-YOLOv5's mAP@50 improves by 1.2 percentage points. In terms of computational cost evaluation metrics, its parameter count, floating-point computation load, and model size are reduced by 37.76%, 40.37%, and 36.81%, respectively. In summary, the key advantage of the improved MSP-YOLOv5 model proposed in this paper is its excellent balance between computational efficiency and detection accuracy. Its lower computational cost and improved detection performance make it ideal for deployment in industrial equipment with high real-time requirements and limited hardware resources.

[0133] 2. Design and optimization of FPGA-based heterogeneous acceleration architecture

[0134] (1) Hardware accelerator system architecture design

[0135] This paper proposes a MSP-YOLOv5 hardware accelerator design based on heterogeneous MPSoC, aiming to achieve efficient target detection tasks. Through hardware and software collaborative design, the solution maps pre-processing, convolution calculation and post-processing to the programmable logic (PL) side and ARM processor side respectively, to achieve a low-latency, high-throughput millimeter wave image hidden object detection system. The hardware accelerator system architecture is as follows: Figure 6 shown.

[0136] The MSP-YOLOv5 hardware accelerator utilizes a heterogeneous computing architecture, comprised of a processing system (PS) and programmable logic (PL). At the system architecture level, the PS is primarily responsible for system scheduling and image post-processing. The PS primarily handles two core functions: system-level task management and image post-processing. Specifically, system-level task management encompasses key operations such as data scheduling, hardware acceleration IP core parameter configuration, and invoking underlying driver libraries. The image post-processing module, running under the Linux operating system, builds a data processing pipeline based on Python. It utilizes high-performance computing libraries such as Numpy and OpenCV to analyze and optimize prediction results, ultimately visualizing object detection results. The programmable logic (PL) is responsible for hardware acceleration of the Letterbox and object detection algorithms. Its parallel implementation of the Letterbox image pre-processing algorithm and the convolutional neural network inference process fully leverages the parallel computing advantages of FPGAs.

[0137] In terms of algorithmic workflow, the system employs a computational task offloading mechanism: the PL side performs hardware acceleration for image preprocessing and forward reasoning, while the PS side is responsible for post-processing tasks such as result decoding and non-maximum suppression (NMS). A heterogeneous communication channel is established between the PS and PL via the AXI high-speed bus, and a DMA transfer mechanism is used to achieve efficient interaction between detection data and image data, ensuring system transmission efficiency. This collaborative hardware and software architecture leverages the FPGA's hardware advantages in streaming data processing and parallel computing through a reasonable division of computational tasks, while retaining the CPU's flexibility in complex logic control and data parsing. It provides a scalable deployment solution for real-time target detection at the edge.

[0138] The overall workflow of the hardware acceleration system is as follows: Figure 7As shown in the figure, ARM will first read the millimeter-wave image stored in the SD card and enable DMA and Letterbox IP cores to pre-process the image. The pre-processed image will be sent to the DPU IP for hidden object detection. ARM will collect the data transmitted by the DPU and perform post-processing operations on it. Finally, the final millimeter-wave hidden object detection results will be visualized on the ARM side.

[0139] (2) System hardware engineering construction

[0140] The clock divider module divides the 100MHz reference clock provided by the ZynqMP processor into multiple signals, providing high-frequency support for the core computing and transmission modules. The 325MHz clock simultaneously drives the DPU convolution computing array, Letterbox IP preprocessing pipeline, AXI-DMA controller, and AXI HP interface, ensuring efficient parallel execution of computing tasks and data transmission. The 650MHz clock is dedicated to the DPU computing unit, which includes the DSP module and requires a clock that is twice the DPU data transmission clock and in the same direction. The low-frequency 100MHz clock is used for the AXI-Lite configuration bus, reset logic, and peripheral control modules.

[0141] The AXI-DMA controller serves as the data transfer core, responsible for high-speed transmission of image preprocessing and inference results. Its MM2S (Memory-to-Stream) channel reads raw data from DDR at a 325MHz clock and pushes it to the Letterbox IP buffer via the AXI4-Stream interface. The S2MM (Stream-to-Memory) channel receives preprocessing results output by the Letterbox IP and writes them back to DDR for processing and analysis by the DPU.

[0142] The ZYNQMP processor serves as the system's control hub, coordinating hardware resources and task scheduling through the Linux driver layer. The software invokes Vitis AI Runtime to load the quantized MSP-YOLOv5 model, dynamically configures the input and output size parameters of the Letterbox IP, and utilizes an interrupt controller to respond to DPU inference completion events and DMA transfer status, enabling pipelined coordinated scheduling of pre-processing, inference, and post-processing tasks. DDR memory is divided into an input image buffer, a pre-processing result area, and a model weight area.

[0143] The AXI interconnect uses a layered crossbar switch architecture to connect the DPU, AXI-DMA, DDR controller, and peripheral modules. At the same time, the AXI4 stream converter is used to adapt to the protocol and bit width differences between different IP cores. For example, the 24-bit pixel stream input format output by Letterbox is aligned to the 32-bit input format of AXI-DMA to match the data transmission bit width. Hardware engineering such as Figure 8 shown.

[0144] (3) Deployment process

[0145] Figure 9 This is a detailed deployment flow chart of the present invention. When hardware acceleration is performed on the model, the overall process can be divided into four parts, namely model construction, hardware platform construction, software program construction, and hardware deployment.

[0146] 3. End-to-end system integration and multi-dimensional performance verification

[0147] (1) Resource occupancy and power consumption analysis

[0148] The hardware accelerator was analyzed using the resource analysis tool in Vivado. The DPU uses the B4096 architecture. Table 3 shows that the final BRAM resource utilization is 82.6%, which is within the controllable resource consumption range of the FPGA.

[0149] Table 3 Accelerator resource usage

[0150]

[0151] The power consumption analysis tool in VIVADO is used to analyze the hardware accelerator. The power consumption analysis results of the accelerator are as follows: Figure 10 As shown in the figure, the total system power consumption is only 6.367W, which is relatively low. This shows that the hardware accelerator performs well in terms of power consumption and is suitable for the low-power deployment requirements of edge devices.

[0152] Looking at the power consumption composition, dynamic power consumption accounts for the vast majority, reaching 6.037W, or 95% of the total. This means that the system's primary power consumption comes from dynamic operations, such as logic operations and signal transmission. In contrast, static power consumption is only 0.330W, accounting for 5% of the total. Static power consumption primarily comes from circuit leakage losses, and this relatively low consumption further demonstrates the hardware accelerator's excellent performance in controlling static power consumption.

[0153] (2) Accelerator detection performance analysis

[0154] This paper quantizes and compiles the MSP-YOLOv5 and YOLOv5s models using the Xilinx VITIS AI development platform, generating an xmodel executable file suitable for FPGAs. The quantized models are then tested for accuracy and speed using a self-built millimeter-wave security inspection image dataset. The experimental results are shown in Table 4.

[0155] Table 4 Comparison of detection results between MSP-YOLOv5 and YOLOv5s on FPGA

[0156]

[0157] (3) Comparison analysis with CPU / GPU

[0158] To further analyze the detection performance of the hardware accelerator studied in this paper, we conducted inference tests on the MSP-YOLOv5 algorithm on both CPU and GPU platforms. This comparative analysis aimed to further evaluate the strengths and weaknesses of the hardware accelerator. During the testing process, mean average detection accuracy (mAP@50%), inference speed (FPS), inference power consumption (W), and energy efficiency ratio (FPS / W) were used as evaluation metrics.

[0159] The hardware platform used for the test is as follows: an Intel(R) Core(TM) i7-12700H CPU and an RTX 3060 GPU. CPU power consumption was collected using the hardware information monitoring tool HWiNFO, while GPU power consumption was obtained using the Nvidia-Smi tool.

[0160] Table 5 shows that compared to a CPU, the hardware accelerator built on an FPGA in this paper achieves an inference speed of 27.3 FPS higher, while reducing power consumption by approximately two-thirds, while only sacrificing 0.9 percentage points in average accuracy. This results in an energy efficiency ratio of approximately 12.95 times that of a CPU. Compared to a GPU, although the detection frame rate of the hardware accelerator built on an FPGA in this paper is only approximately 40% of that of an RTX3060, its power consumption is only 15% of that of a GPU, resulting in an energy efficiency ratio of 2.56 times that of a GPU.

[0161] Table 5 Comparison of model inference performance on different hardware platforms

[0162]

[0163] In resource-constrained edge hardware devices, the FPGA hardware accelerator solution proposed in this invention shows significant advantages. Its high energy efficiency ratio is crucial for millimeter-wave security inspection imaging systems that need to work continuously for a long time. It can not only effectively reduce system energy consumption and reduce operating costs, but also ensure the stability and reliability of the equipment during long-term operation, avoid equipment overheating, failures and other problems caused by excessive energy consumption, thereby ensuring the continuous and efficient implementation of security inspections. In addition, the FPGA hardware accelerator solution also has high precision and low latency characteristics in practical applications. Although its detection accuracy is slightly reduced compared to CPU / GPU, it is within an acceptable range, and the accelerator has excellent frame rate performance, meets real-time requirements, and can complete detection tasks quickly and accurately, providing strong support for security inspections.

Claims

1. A millimeter wave image detection method based on heterogeneous MPSoC, characterized in that: The following steps are involved: The backbone network uses the YOLOv5 algorithm, including the Backbone layer, Neck layer, head layer, and post-processing stage; The Backbone layer preprocesses the millimeter wave image, and the C3-Ghost module is used to replace the original C3 module in the Backbone layer; The Neck layer performs hidden object detection on the preprocessed image, and then enters the SA module of the channel-spatial dual attention mechanism to perform feature channel rearrangement and channel and spatial attention calculation; The head layer performs convolution on the output of the Neck layer and then enters the post-processing stage; In the post-processing stage, the Soft-NMS algorithm is used to replace the traditional NMS, and the Gaussian attenuation function is used to optimize the frame suppression strategy between dense targets; Output millimeter wave hidden object detection results.

2. The millimeter wave image detection method based on heterogeneous MPSoC according to claim 1, characterized in that: The C3-Ghost module adopts a Ghost bottleneck layer, and the Ghost bottleneck layer adopts a dual-module cascade architecture: the first-layer Ghost module increases the channel dimension through a feature expansion mechanism, and the second-layer Ghost module implements channel compression to maintain the consistency of input and output dimensions; feature fusion is achieved between modules through residual connections, and the back end of each sub-module implements batch normalization and ReLU activation function processing in turn.

3. The millimeter wave image detection method based on heterogeneous MPSoC according to claim 2, characterized in that: A depthwise separable convolutional layer is embedded between the two Ghost modules.

4. The millimeter wave image detection method based on heterogeneous MPSoC according to claim 1, characterized in that: The SA module includes the following steps ① Feature grouping For a given feature map X, first divide it into G groups of sub-features along the channel dimension X=[X1,X2,…,X G ],X k ∈R C / G×H×W (1.1) Each group of sub-features X k Further split into two branches X k1 With X k2 , processing channel attention and spatial attention respectively; C represents the number of channels; R C / G×H×W represents a four-dimensional real tensor; ② Channel attention Channel attention generates channel statistics through global average pooling, and then uses a simple gating mechanism with sigmoid activation to generate the final channel attention output. The calculation formulas are shown in (1.2) and (1.3): X' k1 =σ(f c (s))·X' k1 =σ(W1s+b1)·X k1 (1.3) Where W1 and b1 are parameters for scaling and offset, s represents the generated channel statistics, and X' k1 Represents the channel attention output information, H represents the height, W represents the width, i and j represent the index of the feature map in the spatial dimension, f gp represents the global pooling operation, σ represents the activation function, f c (s) represents a fully connected layer; ③Spatial attention Spatial attention generates spatial statistics through group normalization, and then uses a simple gating mechanism with sigmoid activation to generate the final spatial attention output, which is calculated as shown in formula (1.4) X' k2 =σ(W2·GN(X k2 )+b2)·X k2 (1.4) Where W2 and b2 are parameters, X' k2 Represents spatial attention output information; ④ Aggregation and channel reorganization The outputs of the two branches are concatenated along the channel dimension to form X' k =[X' k1 ,X' k2 ], then aggregate all sub-features and promote information interaction between groups through channel shuffling operation, and finally output a feature map with the same size as the input.

5. The millimeter wave image detection method based on heterogeneous MPSoC according to claim 1, characterized in that: The calculation of the Soft-NMS algorithm is shown in formula (1.7): B max represents the detection box with the highest confidence, θ represents the IoU threshold, IOU represents the intersection-over-union ratio between the detection box and the reference box, S i Represents the detection box B j Confidence score of .

6. A millimeter wave image detection system based on heterogeneous MPSoC, characterized by: The system is collaboratively composed of a processor system (PS) and a programmable logic terminal (PL); the backbone layer, the neck layer, and the head layer in the detection method of any one of claims 1 to 5 are mapped to the programmable logic terminal (PL), and the post-processing stage is mapped to the processor system (PS); The processor system (PS) includes an ARM, which reads the millimeter wave image and enables DMA and Letterbox IP cores to pre-process the image; The programmable logic (PL) includes a Letterbox IP core and a DPU IP core. The Letterbox IP core pre-processes the image, and the pre-processed image is sent to the DPU IP core for hidden object detection. The ARM collects the data transmitted by the DPU IP core and decodes and performs post-processing operations on it, and finally visualizes the final millimeter wave hidden object detection results on the ARM side.

7. The millimeter wave image detection system based on heterogeneous MPSoC according to claim 6, characterized in that: A heterogeneous communication channel is established between the PS and PL via an AXI high-speed bus, and a DMA transmission mechanism is used to achieve efficient interaction between detection data and image data.

8. The millimeter wave image detection system based on heterogeneous MPSoC according to claim 6, characterized in that: The processor system (PS) also performs data scheduling, parameter configuration of the hardware acceleration IP core, and calling of the underlying driver library; under the Linux operating system environment, the ARM builds a data processing pipeline based on the Python language, and implements the analysis and optimization of the prediction results by calling Numpy and OpenCV, and finally completes the visualization of the target detection results.

9. The millimeter wave image detection system based on heterogeneous MPSoC according to claim 6, characterized in that: The programmable logic terminal (PL) further includes a ZYNQMP processor, a clock frequency division module, an AXI-DMA controller and an AXI HP interface; The clock divider module divides the 100MHz reference clock provided by the ZynqMP processor into multiple signals. The 325MHz clock simultaneously drives the convolution calculation array of the DPU IP core, the Letterbox IP core pre-processing pipeline, the AXI-DMA controller, and the AXI HP interface; the 650MHz clock is used for the calculation unit of the DPU IP core; and the 100MHz clock is used for the AXI-Lite configuration bus, reset logic, and peripheral control module. The AXI-DMA controller is used for high-speed transmission of image preprocessing and inference results. Its MM2S channel reads raw data from DDR at a 325MHz clock and pushes it to the Letterbox IP core buffer via the AXI4-Stream interface. The S2MM channel receives the preprocessing results output by the Letterbox IP core and writes them back to DDR for processing and analysis by the DPU. The ZYNQMP processor, serving as the system control hub, coordinates hardware resources and task scheduling through the Linux driver layer. The software side calls Vitis AI Runtime to load the quantized MSP-YOLOv5 model, dynamically configures the input and output size parameters of the Letterbox IP core, and uses the interrupt controller to respond to the DPU IP core's inference completion events and DMA transfer status, enabling pipelined coordinated scheduling of pre-processing, inference, and post-processing tasks. The AXI interconnect uses a hierarchical crossbar switch architecture to connect the DPU, AXI-DMA controller, DDR controller, and peripheral modules.

10. The millimeter wave image detection system based on heterogeneous MPSoC according to claim 6, characterized in that: The processor system (PS) also includes a DDR controller, and the DDR storage space is divided into an input image buffer area, a preprocessing result area and a model weight area.

Images