A lightweight millimeter wave radar two-dimensional feature map classification method and system for FPGA hardware deployment

Abstract

The application provides a lightweight millimeter wave radar two-dimensional feature map classification method and system for FPGA hardware deployment, and solves the technical problems of poor real-time performance and high power consumption of existing millimeter wave radar two-dimensional feature map classification methods in edge computing scenarios. It includes obtaining a feature map and preprocessing it to obtain a preprocessed feature map, constructing a dataset, and dividing the dataset into a training set and a validation set; building a classification network model based on LeNet, training the classification network model using the training set, and obtaining a trained classification network model; verifying whether the performance of the trained classification network model meets the performance indicators using the validation set, and if so, obtaining a validated classification network model, deploying it to an FPGA, classifying the feature map based on the classification network model, and obtaining a classification result; otherwise, continue training to update the parameters of the classification network model until the performance after verification meets the performance indicators. The application can be widely applied in the field of image classification technology.

Description

A lightweight two-dimensional feature map classification method and system for millimeter-wave radar deployed on FPGA hardware Technical Field

[0001] This application belongs to the field of image classification technology, and more specifically, it relates to a lightweight millimeter-wave radar two-dimensional feature map classification method and system for FPGA hardware deployment. Background Technology

[0002] Millimeter-wave radar, with its advantages of strong anti-interference capability and good environmental adaptability, has been widely used in edge computing scenarios such as autonomous driving, intelligent security, and mobile terminals. Among them, two-dimensional feature map classification is the core link of millimeter-wave radar data processing, and its processing efficiency directly determines the response speed and decision-making effectiveness of the radar system.

[0003] However, in real-time decision-making edge computing scenarios such as obstacle classification, gesture classification, and posture classification for autonomous driving, existing two-dimensional feature map classification methods are mostly based on serial computing modes of general-purpose processor architectures. These methods suffer from high computational complexity and processing latency, making it difficult to meet the urgent demands for low latency and high real-time performance in scenarios such as emergency obstacle avoidance by autonomous vehicles, rapid response to gesture commands in human-computer interaction, and dynamic human posture recognition. Furthermore, edge hardware such as mobile terminals and embedded devices are generally limited by limited computing and power resources. Existing classification methods often consume significant computing resources and power, which not only exacerbates energy consumption but may also lead to decreased system stability due to excessive hardware resource usage, making them unsuitable for the low-power operation requirements of edge devices. Summary of the Invention

[0004] The purpose of this application is to provide a lightweight millimeter-wave radar two-dimensional feature map classification method and system for FPGA hardware deployment, so as to solve the technical problems of poor real-time performance and high power consumption of existing millimeter-wave radar two-dimensional feature map classification methods in edge computing scenarios.

[0005] To achieve the above objectives, the first embodiment of this application provides a lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment, including the following steps:

[0006] Obtain feature maps and perform preprocessing to obtain preprocessed feature maps. Construct a dataset and divide the dataset into a training set and a validation set.

[0007] A classification network model is built based on LeNet, and the training set is used to train the classification network model to obtain a trained classification network model.

[0008] The performance of the trained classification network model is verified using a validation set to determine if it meets the performance metrics. If it does, the validated classification network model is obtained and deployed to the FPGA. The feature maps are then classified based on the classification network model to obtain the classification results. Otherwise, training continues to update the parameters of the classification network model until the validated performance meets the performance metrics.

[0009] Preferably, the classification network model includes an input layer, a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a channel expansion projection layer, a global max pooling layer, and a fully connected layer;

[0010] The preprocessed feature map is input into the input layer. After convolution in the first convolutional layer, the feature map after the first convolution is obtained. Then, it is input into the first max pooling layer for pooling, resulting in the feature map after the first pooling. Next, it is input into the second convolutional layer for convolution, resulting in the feature map after the second convolution. Finally, it is input into the second max pooling layer for pooling, resulting in the feature map after the second pooling. Then, it is input into the channel expansion projection layer for linear transformation, resulting in the projected feature map. Finally, it is input into the global max pooling layer for pooling, resulting in the feature vector. Finally, it is input into the fully connected layer for linear transformation, resulting in the classification result.

[0011] Preferably, during the deployment of the first and second convolutional layers onto the FPGA, a dual-parallel architecture is set to compute the product of the preprocessed feature map and the weight parameters.

[0012] The dual-parallel architecture is based on column-parallel data prefetching. The preprocessed feature map is synchronously latched according to the column size of the convolution kernel to complete parallel loading. By deploying a dynamic data rearrangement network, the serially input pixel column blocks are reconstructed into a spatially parallel pixel array according to the sliding window time sequence. The reconstructed pixel array and weight parameters are spatially aligned through a multiplication array. Synchronous fixed-point multiplication is performed on the pixels at the corresponding positions to obtain the product.

[0013] Preferably, during the deployment of the first and second max pooling layers onto the FPGA, a comparison architecture based on a two-row pipeline is set up. When the even-numbered rows of the feature map after the first convolution are input, the maximum value within the window of adjacent pixels in the same pooling window within the row is extracted by a parallel comparator to obtain the local maximum value of the even-numbered rows. When the odd-numbered rows are input, the comparison architecture compares the pixel pairs within the window of the odd-numbered rows in real time to obtain the local maximum value of the odd-numbered rows. At the same time, it reads the local maximum value of the even-numbered rows that matches the current odd-numbered row and column coordinates, and compares them to determine the maximum value of the first and second max pooling layers.

[0014] Preferably, during the deployment of the channel expansion projection layer and the global max pooling layer, the channel expansion projection layer and the global max pooling layer are cascaded, and channel expansion is performed through several convolutional kernels to expand the number of channels in the feature map after the second pooling. The spatial dimension of the feature map after the second pooling is compressed through the global max pooling layer.

[0015] Preferably, in an FPGA, a softmax function with omitting exponential normalization is deployed at the output of the fully connected layer, and the maximum value index is selected as the classification result by comparing the numerical values ​​of the output vectors.

[0016] Preferably, the validation set is input into the trained classification network model to perform forward propagation, the classification result predicted by the trained classification network model is calculated, and it is compared with the real classification result to evaluate the generalization ability of the trained classification network model.

[0017] The forward propagation process is reproduced and verified in FPGA to determine whether the trained classification network model meets the performance indicators.

[0018] Preferably, the process of extracting the maximum value within the window using a parallel comparator includes: comparing adjacent pixels in the column coordinates of the even-numbered rows of the data stream, and temporarily storing the local maximum value of the even-numbered rows in a FIFO.

[0019] Preferably, preprocessing refers to performing grayscale binarization on the feature map and rotating it counterclockwise by 90° to obtain the preprocessed feature map.

[0020] The second embodiment of this application provides a lightweight millimeter-wave radar two-dimensional feature map classification system for FPGA hardware deployment, including: a preprocessing module, a model building module, a verification module, and a classification module;

[0021] The preprocessing module is used to acquire feature maps and perform preprocessing to obtain preprocessed feature maps, construct datasets, and divide the datasets into training sets and validation sets.

[0022] The model building module is used to build a classification network model based on LeNet, and to train the classification network model using the training set to obtain the trained classification network model.

[0023] The validation module is used to verify whether the performance of the trained classification network model meets the performance index using the validation set. If it does, the validated classification network model is obtained and deployed to the FPGA; otherwise, training continues to update the parameters of the classification network model until the validated performance meets the performance index.

[0024] The classification module uses a classification network model deployed in the FPGA to classify the feature maps and obtain the classification results.

[0025] The beneficial effects of this application are as follows: This application provides a lightweight millimeter-wave radar two-dimensional feature map classification method and system for FPGA hardware deployment. First, feature maps are acquired and preprocessed to construct a dataset, which is then divided into training and validation sets, providing a data foundation for the subsequent training and validation of the classification network model. Next, a classification network model with fewer parameters and lower complexity is built based on LeNet. The first fully connected layer in the traditional LeNet is replaced with a channel expansion projection layer and a global max pooling layer. Through the cascaded design of the channel expansion projection layer and the global max pooling layer, sparse and low-rank data representation is provided for hardware deployment. This reduces the computational load of inference while suppressing overfitting, ensuring classification accuracy and providing lightweight model support for low-computing-power, low-power operation of edge devices. Finally, the performance metrics of the model are verified using the validation set to ensure that the model meets performance standards in the hardware environment and avoids misclassification caused by insufficient model performance. Finally, the validated classification network model was deployed on an FPGA. Hardware accelerator architectures such as dual parallel architecture and comparison architecture based on dual-row pipeline were designed to address the computational characteristics of each layer of the network. Combined with hardware-friendly algorithms such as data flow optimization scheduling and fixed-point quantization, the inference speed was significantly improved and the system power consumption was reduced while maintaining high classification accuracy, thus completing the intelligent sensing application of millimeter-wave radar on an embedded platform. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 is a schematic diagram of the overall process of a lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment provided in an embodiment of this application.

[0028] Figure 2 is a schematic diagram of the layout of each level of the classification network model provided in an embodiment of this application;

[0029] Figure 3 is a schematic diagram of the module division of the hardware deployment of the classification network model provided in an embodiment of this application;

[0030] Figure 4 is a schematic diagram of the feature map pixel data and RAM mapping mechanism in the first cache module provided in an embodiment of this application;

[0031] Figure 5 is a simulation waveform diagram of the first cache module provided in an embodiment of this application;

[0032] Figure 6 is a schematic diagram of the hardware implementation of the first convolutional layer module provided in an embodiment of this application;

[0033] Figure 7 is a schematic diagram of a dual-parallel architecture provided in an embodiment of this application;

[0034] Figure 8 is a simulation output waveform diagram of the first convolutional layer module provided in an embodiment of this application;

[0035] Figure 9 is a numerical tensor diagram of the output of the first convolutional layer module during the forward propagation process provided in an embodiment of this application;

[0036] Figure 10 is a schematic diagram of a comparison architecture based on dual-row pipeline provided in an embodiment of this application;

[0037] Figure 11 is a schematic diagram of the data flow of a comparison architecture based on two-row pipelines provided in an embodiment of this application;

[0038] Figure 12 is a simulation output waveform diagram of the first maximum pooling layer module provided in an embodiment of this application;

[0039] Figure 13 is a numerical tensor diagram of the output of the first maximum pooling layer module during the forward propagation process provided in an embodiment of this application;

[0040] Figure 14 is a schematic diagram of a channel-level spatial parallel architecture provided in an embodiment of this application;

[0041] Figure 15 is a simulation output waveform diagram of the second convolutional layer module provided in an embodiment of this application;

[0042] Figure 16 is a numerical tensor diagram of the output of the second convolutional layer module during the forward propagation process provided in an embodiment of this application;

[0043] Figure 17 is a simulation output waveform diagram of the second maximum pooling layer module provided in an embodiment of this application;

[0044] Figure 18 is a numerical tensor diagram of the output of the second maximum pooling layer module during the forward propagation process provided in an embodiment of this application;

[0045] Figure 19 is a schematic diagram of the structure of the channel expansion projection layer and the global max pooling layer cascade optimization strategy provided in an embodiment of this application;

[0046] Figure 20 is a bar chart of system resource utilization corresponding to the first fully connected layer of the original LeNet model provided in an embodiment of this application;

[0047] Figure 21 is a bar chart of system resource utilization corresponding to the cascaded optimization strategy of channel expansion projection layer and global maximum pooling layer provided in an embodiment of this application;

[0048] Figure 22 is a hardware architecture diagram of a channel extension projection layer and a global maximum pooling layer cascade module provided in an embodiment of this application;

[0049] Figure 23 is a simulation output waveform diagram of the channel extended projection layer module provided in an embodiment of this application;

[0050] Figure 24 is a simulation output waveform diagram of a global maximum pooling layer module provided in an embodiment of this application;

[0051] Figure 25 is a numerical tensor diagram of the output of the channel expansion projection layer and the global maximum pooling layer cascade module during the forward propagation process provided in an embodiment of this application;

[0052] Figure 26 is a schematic diagram of the hardware structure of a fully connected layer module provided in an embodiment of this application;

[0053] Figure 27 is a simulation output diagram of a first type of fully connected layer module provided in an embodiment of this application;

[0054] Figure 28 is a simulation output diagram of a second type of fully connected layer module provided in an embodiment of this application;

[0055] Figure 29 is a simulation output diagram of a third type of fully connected layer module provided in an embodiment of this application;

[0056] Figure 30 is a simulation output diagram of a fourth type of fully connected layer module provided in an embodiment of this application;

[0057] Figure 31 is a simulation output diagram of the fifth type of fully connected layer module provided in an embodiment of this application;

[0058] Figure 32 is a simulation output diagram of the sixth type of fully connected layer module provided in an embodiment of this application;

[0059] Figure 33 is a numerical tensor diagram of six categories output by the fully connected layer module during the forward propagation process according to an embodiment of this application;

[0060] Figure 34 is a schematic diagram of the data flow of the softmax function provided in an embodiment of this application;

[0061] Figure 35 is a simulation output waveform of the softmax function provided in an embodiment of this application;

[0062] Figure 36 is a schematic diagram of the final output classification result of forward propagation verification provided in an embodiment of this application;

[0063] Figure 37 is a waveform diagram of classification delay during FPGA testing according to an embodiment of this application;

[0064] Figure 38 is a schematic diagram of classification delay in forward propagation verification provided in an embodiment of this application. Detailed Implementation

[0065] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this application.

[0066] Please refer to Figure 1, which illustrates a lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment provided in the first embodiment of this application, including:

[0067] S1: Obtain feature maps and perform preprocessing to obtain preprocessed feature maps. Construct a dataset and divide the dataset into training and validation sets.

[0068] Two-dimensional feature maps are acquired using millimeter-wave radar, and preprocessed to obtain preprocessed feature maps. A dataset is constructed, and the preprocessed feature maps are randomly divided according to a preset ratio to obtain training and validation sets. Preprocessing refers to performing grayscale binarization on the feature maps and rotating them counterclockwise by 90° to obtain the preprocessed feature maps. This application does not limit the type of feature map; two-dimensional feature maps such as micro-Doppler maps, range Doppler maps, and range-angle maps can be selected according to the actual situation.

[0069] In an optional embodiment, an AWR1642 millimeter-wave radar is used to acquire six categories (different postures) of micro-Doppler images, with an initial size of 256×64. After grayscale binarization of the acquired micro-Doppler images, these feature maps are rotated 90° counterclockwise to obtain a preprocessed feature map with a size of 64×256. This application rotates the feature map 90° counterclockwise to ensure the compatibility of the software and hardware integration during subsequent training, achieving software and hardware collaboration. The above description uses six categories (different postures) of micro-Doppler images as an example, but the categories are not limited to posture and can be extended to other classification and recognition scenarios such as gestures and obstacles.

[0070] During the feature map acquisition phase, labels (label numbers 1-6) were first assigned to each of the six categories to ensure that the category identifier of each feature map was unique and distinguishable. Eight subjects of varying heights and builds were selected, and 10 sets of data were collected from each subject for each category. For each set, sampling points at different fixed locations were extracted to generate 16 feature maps, resulting in 1280 feature maps for each category, for a total of 7680 feature map samples across the six categories. These samples were then preprocessed to obtain preprocessed feature maps. 80% of the preprocessed feature maps were randomly selected from each set as the training set, and the remaining 20% ​​was used as the validation set.

[0071] S2: Build a classification network model based on LeNet, and train the classification network model using the training set to obtain a trained classification network model.

[0072] A classification network model is built based on the basic form of LeNet, with a single-channel 64×256 feature map as input. The classification network model includes an input layer, a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a channel expansion projection layer, a global max pooling layer, and a fully connected layer. The specific parameters of the model are shown in Table 1.

[0073] Table 1: Specific parameter settings for each level of the classification network model

[0074]

[0075] The training set (64×256 feature maps) was input into the pre-built classification network model for training, resulting in a trained classification network model. The parameters (weights w and biases b) of each layer of the model were stored. The model was trained for 50 epochs, and the classification accuracy of the model after training was 98.96%. The equipment parameters used during training were as follows: Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz, 16GB RAM, NVIDIA GeForce RTX 2060 GPU (14GB VRAM), Windows 10 operating system, and PyTorch 1.9.0, Python 3.8 (Ubuntu 18.04 environment), and CUDA 11.1.

[0076] S3: Use the validation set to verify whether the performance of the trained classification network model meets the performance indicators. If yes, obtain the validated classification network model, deploy it in the FPGA, and classify the feature map based on the classification network model to obtain the classification result; otherwise, continue training to update the parameters of the classification network model until the validated performance meets the performance indicators.

[0077] The trained classification network model is tested via forward propagation to verify its performance. Specifically, the forward propagation process on the validation set is as follows:

[0078] Please refer to Figure 2. The input validation set (feature map of size [1, 64, 256]) is first convolved by the first convolutional layer (kernel size 3×3, stride 1, padding 0) and the ReLU activation function is applied, outputting a feature map of size [8, 62, 254] after the first convolution; then it is pooled by the first max pooling layer (kernel size 2×2, stride 2), outputting a feature map of size [8, 31, 127] after the first pooling.

[0079] The feature map after the first pooling is convolved by the second convolutional layer (kernel size 3×3, stride 1, padding 0) and the ReLU activation function is applied, outputting a feature map after the second convolution with size [8, 29, 125]. Then, it is pooled by the second max pooling layer (kernel size 2×2, stride 2), outputting a feature map after the second pooling with size [8, 14, 62].

[0080] The feature map after the second pooling is linearly transformed by the channel expansion projection layer (kernel size 1×1, stride 1, padding 0) and the ReLU activation function is applied to output a projected feature map with size [100, 14, 62]. Then it is passed through the global max pooling layer to output a feature vector with size

[100] .

[0081] Finally, the feature vector is linearly transformed through a fully connected layer to output a feature vector of size [6], thus obtaining the classification result.

[0082] An independent validation set is input into the trained classification network model to perform forward propagation. The model's classification results predicted by the validation set are compared with the labels of the true classification results to evaluate the model's generalization performance (such as accuracy, precision, recall, F1 score, loss value, etc.), providing data support for model tuning or deployment decisions. Furthermore, the forward propagation process (including its specific computational graph structure, activation function, quantization scheme, etc.) is reproduced and validated on the target FPGA hardware platform. This reproduction and validation confirms whether the model meets performance metrics in the FPGA environment. Specifically, it confirms the correctness of the model's functionality, whether the computational accuracy meets expectations, and whether the data processing latency meets constraints in the FPGA environment, laying a solid foundation for subsequent model deployment and integration on the FPGA hardware platform.

[0083] Furthermore, to achieve efficient inference of the validated classification network model in embedded scenarios, this application uses an FPGA (Field-Programmable Gate Array) as the target hardware platform for deployment. The core architecture of the deployment scheme is shown in Figure 3, constructing a modular hardware processing pipeline, including a first cache module, a first convolutional layer module, a first max-pooling layer module, a second cache module, a second convolutional layer module, a second max-pooling layer module, a cascaded module of channel extension projection layer and global max-pooling layer, a fully connected module, and a softmax function. The modules are tightly coupled and interact with each other through a high-speed, low-latency on-chip interconnect structure. The collaborative work between modules strictly follows a data flow-driven mechanism, ensuring that the input feature map, after preprocessing by the radar signal preprocessing system, can efficiently flow through each module and each level of the computing engine for parallel acceleration. The calculation results are orderly transmitted and output as classification results with the support of the on-chip caching mechanism. This architecture design aims to fully utilize the parallel computing capabilities and reconfigurability of the FPGA to achieve high-throughput, low-latency inference of the model while meeting stringent resource constraints, providing hardware acceleration support for classification and recognition applications.

[0084] Specifically, referring to Figure 4, the first cache module receives and temporarily stores the feature map preprocessed by the radar signal preprocessing system. To accommodate efficient storage and parallel access, its core storage unit adopts a distributed architecture, consisting of 64 on-chip memory (RAM) instances with a depth of 256. This architecture uses a row-addressing driven mode, meaning that each row of 256 pixels in the input feature map is directionally latched into an independent RAM block corresponding to that row number. This architecture ensures that during the feature map writing phase, the corresponding RAM enable signal is activated only when the data in a specific row is valid, thereby achieving precise row-level storage control and efficient resource utilization. The first cache module integrates status monitoring logic to continuously track the write status of each RAM. Once it is detected that all 64 rows of image data (i.e., a complete feature map of size 64×256) have been successfully cached into their respective designated RAM units, this logic will pull the conv_start control signal high. The simulation waveform of this module is shown in Figure 5. The high signal indicates that the current feature map is ready and the first convolutional layer module can be started for calculation, avoiding the risk of erroneous operation when the data is not fully cached, and ensuring that the calculation unit can obtain a complete and aligned feature map in a timely manner.

[0085] Furthermore, the first convolutional layer module integrates eight 3×3 convolutional kernels and performs sliding window convolution operations with a stride of 1. Its data processing flow is as follows: it receives a preprocessed feature map with a single-channel size of 64×256, performs convolution operations, and outputs an 8-channel feature map with a size of 62×254 after the first convolution. To optimize hardware computational efficiency, this module adopts a pre-scaling fixed-point representation strategy: the weight parameters of the original floating-point convolutional kernels are uniformly magnified by a factor of 8192 and then rounded to convert them into fixed-point representations, significantly reducing the computational complexity of subsequent multiply-accumulate (MAC) units.

[0086] Referring to Figure 6, the weight storage architecture adopts a row-distributed ROM configuration: the weight parameters of the first row of the 8 convolutional kernels are integrated and stored in ROM0, the second row in ROM1, and the third row in ROM2. Simultaneously, the 8 bias parameters are independently stored in ROM_b. Accordingly, the module deploys a total of 3 ROM kernels with a depth of 24 to store the weight parameters and 1 ROM kernel with a depth of 8 to store the bias parameters. During computation, the preprocessed feature map input is spatially multiplied and accumulated with the weight parameters and bias parameters in the ROM through a sliding window mechanism (the preprocessed feature map is multiplied with the weight parameters in the ROM and then the bias parameters are accumulated), ultimately outputting an 8-channel feature map with a scale of 62×254 after the first convolution.

[0087] In the hardware implementation of the first convolutional layer module, this application proposes a dual-parallel architecture for calculating the product of the preprocessed feature map and the weight parameters. The convolutional computation unit adopts a two-stage pipelined parallel architecture to optimize throughput.

[0088] Please refer to Figure 7. The left side shows the logical flow of the dual-parallel architecture, and the right side shows the temporal execution flow. The first stage is parallel reading. Based on column-parallel data prefetching, the preprocessed feature map is synchronously latched according to the column size of the convolution kernel (3 pixels, corresponding to D1-D3 in Figure 7), realizing parallel loading of window-level data granularity. The second stage is parallel arrangement. By deploying a dynamic data rearrangement network, the serially input pixel columns are reconstructed into a spatially parallel pixel array according to the sliding window temporal sequence, ensuring that a complete 3×3 convolution window data can be output in each clock cycle. The reconstructed pixel array and the pre-stored weight parameters of the convolution kernel (P00, P01, P02, P10, P11, P12, P20, P21, P22) are spatially aligned through a fully connected multiplication array. The pixels at corresponding positions of the two perform synchronous fixed-point multiplication operations, generating 9 product results in parallel within a single cycle.

[0089] In the timing execution flow of the dual-parallel architecture on the right side of Figure 7, P00, P01, P02, P10, P11, P12, P20, P21, and P22 in signals para_h0, para_h1, and para_h2 represent the weights of the convolution kernel in the ROM core. For example, P00, P01, and P02 are the three weights in the first row of the convolution kernel, P10, P11, and P12 are the three weights in the second row of the convolution kernel, and so on. Signals data_rd_arry_d0, data_rd_arry_d1, and data_rd_arry_d2 represent the output after parallel reading and parallel arrangement. Among them, the areas marked by the red box: D2, D3, and D4 in data_rd_arry_d0; D1, D2, and D3 in data_rd_arry_d1; and D0, D1, and D2 in data_rd_arry_d2 represent the 3×3 window data within a certain clock cycle. D5 in data_rd_arry_d0, D4 in data_rd_arry_d1, and D3 in data_rd_arry_d2 represent the data for the next sliding window, demonstrating that the hardware architecture of this application operates in a pipeline manner. This indicates that the architecture can output a continuous spatial parallel data array, showcasing the high throughput of the dual-parallel architecture.

[0090] Through the aforementioned dual-parallel architecture, the system can complete a full convolutional window calculation in each clock cycle. After the serial data stream is transformed into parallel window data through the rearranged network, the multiplication array outputs the product of all pixels within the window and the kernel weights in real time. This architecture reduces the O(n²) computational complexity of traditional serial convolution to O(1) level window processing latency, requiring only two pipelined operations to complete the window calculation, significantly reducing the overall convolutional layer latency.

[0091] The simulation output waveform of the first convolutional layer module (Figure 8) was compared with the parameters of the forward propagation verification network built in PyCharm (Figure 9) to verify the completeness of the module's function. The data from Figures 8 and 9 show that the core numerical imaging is highly consistent, with only a slight deviation. This difference is due to the quantization error caused by the hardware pre-scaling fixed-point strategy (floating-point weights amplified by 8192 times and rounded down), which is a reasonable deviation within the expected range. This indicates that the module's function meets design expectations, and the computational accuracy satisfies the requirements.

[0092] Furthermore, in the hardware implementation of the first max pooling layer module, a 2×2 downsampling kernel is used to process the 8-channel, 62×254-sized feature map output from the previous module after the first convolution, resulting in an 8-channel, 31×127-sized feature map after the first pooling. Referring to Figure 10, this application proposes a comparison architecture based on a two-row pipeline. When the even-numbered rows of the feature map (index starting from 0, D00, D01, D02, ...) data stream is input, the maximum value within the window is extracted for adjacent pixels belonging to the same 2×2 pooling window in that row through a parallel comparator (a>b). That is, the adjacent pixel pairs with column coordinates of 2n and 2n+1 in the even-numbered row data stream are compared (e.g., D00 is compared with D01). The resulting local maximum value of the even-numbered row is temporarily stored in a dedicated row buffer FIFO with a depth of 127. The address of this FIFO corresponds one-to-one with the column coordinates of the feature map. Simultaneously, when subsequent odd-numbered rows of data arrive, the module performs two operations: first, it compares pixel pairs within the odd-numbered row window in real time (e.g., comparing D10 and D11) to obtain the local maximum value of the odd-numbered row; second, it reads the local maximum value of the even-numbered row that matches the current odd-numbered row and column coordinates from the FIFO, and performs a final comparison between the two to determine the maximum value of the 2×2 pooling window.

[0093] Throughout the process, the read and write addresses of the FIFO are strictly aligned with the coordinates of the feature map columns to avoid data misalignment. The FIFO with a depth of 127 precisely matches the dimension of the feature map columns, which significantly improves the utilization of hardware resources while eliminating data conflicts between rows. Finally, a pooling result is output in a pipelined manner every cycle, completing the downsampling from 62×254 to 31×127.

[0094] In an optional embodiment, please refer to Figure 11, which is a timing flowchart based on a two-row pipelined comparison architecture. When an even-numbered row of the feature map (D00, D01, D02, D03... in act_data) is input, a parallel comparator is used to perform in-window comparison of adjacent pixels in that row (as shown in the red box in Figure 11, the ratio of D01 to D00, and the ratio of D03 to D02), extracting the local maximum value of the even-numbered row (D00, D03 in wr_data), and writing it into a row buffer FIFO with a depth of 127 through the enable signal wr_en, ensuring that the write address is aligned with the column coordinates after pooling.

[0095] When odd-numbered rows of data (D10, D11, D12, D13, etc. in act_data) arrive, the parallel comparator first performs an in-window comparison of adjacent pixels in that row (e.g., comparing D10 with D11) to obtain the local maximum value of the odd-numbered row (D10 in Figure 11). At the same time, the even-numbered row buffer values ​​at the corresponding column positions are read from the FIFO through the enable signal rd_en (D00, D03 in rd_data in Figure 11).

[0096] The local maximum value (D10) of the odd-numbered rows is compared with the local maximum value of the even-numbered rows read from the FIFO in the parallel comparator to obtain the maximum value of the 2×2 pooling window (D10 and D03 in pool_data in Figure 11). Finally, the feature map after the first pooling is output through the pool_en signal.

[0097] The simulation output waveform of the first max pooling layer module (Figure 12) is compared with the parameters of the forward propagation verification network built in PyCharm (Figure 13) to verify the integrity of the module's function. As can be seen from the figure, the core pooling results of the two are consistent, indicating that the data are consistent and that the module is functioning normally.

[0098] Furthermore, the second caching module is used to cache the data between the first max pooling layer module and the second convolutional layer module. After caching 8 feature maps of size 31×127, the conv2_start signal is pulled high for one clock cycle to start the second convolutional layer.

[0099] Further, referring to Figure 14, the second convolutional module integrates eight 8×3×3 three-dimensional convolutional kernels (corresponding to eight input channels and eight output channels), performing sliding convolution operations with a stride of 1 to reduce the dimensionality of the input 8-channel, 31×127 feature map after the first pooling to 8 channels, 29×125. This application proposes a channel-level spatial dual-parallel architecture, which can maximize FPGA resource efficiency, synchronously instantiate eight independent convolutional modules, each module is specifically responsible for the computation task of a single output channel, and each module accesses the feature map after the first pooling of the eight input channels cached in the second cache module in parallel. By adopting a dual-parallel architecture, cross-channel computation parallelization is achieved. That is, when the traditional solution requires eight times the time to complete multi-channel convolution, this architecture achieves synchronous generation of eight-channel outputs through a hardware replication strategy, making the processing time of a single feature map equivalent to the parallel computation cycle of eight feature maps, and theoretically improving the throughput by eight times.

[0100] The simulation output waveform of the second convolutional layer module (Figure 15) was compared with the forward propagation verification network built in PyCharm (Figure 16) to verify the integrity of the module's function. The data of the two are consistent, indicating that the module is functioning normally.

[0101] Furthermore, the second max pooling layer module reduces the size of the 8 feature maps (29×125) output by the second convolutional layer module by a factor of 8192, performs max pooling, and then crops the last row and last column of the feature maps to obtain 8 feature maps (28×124). Finally, it outputs 8 feature maps (14×62).

[0102] The simulation output waveform of the second max pooling layer module (Figure 17) was compared with the parameters of the forward propagation verification network built in PyCharm (Figure 18) to verify the integrity of the module's function. The data of the two are consistent, indicating that the module is functioning normally.

[0103] Furthermore, referring to Figure 19, this application improves the fully connected layer in the original LeNet model with an optimization strategy that cascades a channel-expanded projection layer and a global max-pooling layer. This addresses both the overfitting problem and the issue of parameter inflation in the fully connected layer during model deployment. In the network architecture of this application, the parameters of the first fully connected layer module in the original LeNet model account for 99.8% of the total network parameters. This high-dimensional weight matrix not only leads to a surge in computational complexity but also causes overfitting due to the dramatic increase in model degrees of freedom. To solve these problems, this application adopts a cascaded optimization strategy of a channel-expanded projection layer and global max-pooling, firstly by... indivual The convolution kernel performs channel expansion, increasing the number of channels in the feature map from... Expand to Then, a global max pooling layer is used to convert the spatial dimension to... The feature maps are compressed to 1×1. This cascaded optimization strategy reduces the input feature dimension of the first fully connected layer of the original LeNet model from 1×1. Down to It provides sparse, low-rank data representation for hardware deployment while suppressing overfitting.

[0104] Calculate the number of parameters in the first fully connected layer of the original LeNet model. The formula is as follows:

[0105] ;

[0106] In the formula, Input the number of channels. The height of the feature map, The width of the feature map, This is the number of output nodes.

[0107] In one alternative embodiment, , , The number of parameters in the first fully connected layer of the original LeNet model can be calculated. .

[0108] Calculate the parameters of the cascaded optimization strategy of the channel extension projection layer and global max pooling after the replacement. The formula is as follows:

[0109] ;

[0110] In the formula, Input the number of channels. This represents the number of output channels.

[0111] In one alternative embodiment, , The computational cascade optimization strategy can be obtained ,and In comparison, the number of parameters decreased by 99.8%, solving the overfitting problem of the model (the accuracy during training was 100%), and finally the accuracy of the entire lightweight millimeter-wave radar 2D feature map classification network model reached 98.6%. At the same time, after replacing the first fully connected layer with a channel expansion projection layer and a global max pooling cascade optimization strategy, the original block memory resource consumption was reduced from 98% to 9%. Therefore, this optimization strategy can be deployed on a low-cost FPGA, as shown in Figures 20 and 21.

[0112] In the hardware implementation, the input feature map after the second pooling (8×14×62) is first precisely cached through eight 868-block RAMs. Then, nine parameter memories are configured simultaneously, including eight 100-block ROMs (ROM_0-ROM_7) that store the weight parameters of the 1×1 convolution kernel according to the output channel index (each ROM contains 100 sets of 8-dimensional weight vectors), and another 100-block ROM_b that stores the bias parameters. During the computation phase, parallel convolution is adopted, that is, the same address of the eight weight ROMs is activated simultaneously in each cycle, which is equivalent to a convolution kernel. Spatially aligned parallel multiplication and accumulation operations are performed with the 8-channel feature map. This process compresses the 8-channel feature map into a single-channel feature map through cross-channel accumulation. Then, the channels are expanded to output a 100-channel feature map. Finally, the 14×62 feature map is compressed into 1×1 nodes through a global max pooling layer, resulting in a total of 100 nodes, as shown in Figure 22.

[0113] The simulation output waveform of this module (Figures 23-24) is compared with the forward propagation verification network built in PyCharm (Figure 25) to verify the completeness of the module's function. The data of the two are consistent, indicating that the module functions well.

[0114] Furthermore, the fully connected layer module serves as a general feature projection operator in the model, using a high-dimensional (M×N) weight matrix. With bias vector Implement input features To output features The affine transformation. The formula for the affine transformation is: Its computational essence is a mapping of the inner product space between the input vector and the weight vector, that is, for each output node. , characterizing the linear combination strength of input features in different directions, where, For the first The input feature and the first The connection weights of each output feature For the first Each input feature For the first A bias vector. Fully connected layer modules break the local receptive field constraints of convolutional layers through global parameter connections, giving the model the ability to integrate global features. Under the action of a non-linear activation function (such as ReLU), the final output will pass through... The nonlinear transformation learns the nonlinear boundary of the input space and ultimately outputs feature vectors (Softmax) in classification tasks or generates continuous predicted values ​​in regression tasks.

[0115] Referring to Figure 26, in the hardware-implemented fully connected layer module, the 100 nodes output from the previous stage are cached by a distributed RAM with a depth of 100, whose storage depth precisely matches the dimension of the input nodes; a weight ROM with a depth of 600 and a bias ROM with a depth of 6 are configured synchronously. During the computation phase, block multiplication and accumulation optimization is performed, that is, the 600 weights are divided into 6 100-dimensional weight blocks, and then multiplied and accumulated by the 100 nodes in 6 groups, resulting in a total of 6 nodes.

[0116] The simulation output waveforms of the six nodes of this module (Figures 27-32) are compared with the forward propagation verification network built in PyCharm (Figure 33) to verify the integrity of the module's function. The data trends of the two are consistent, indicating that the module is functioning normally.

[0117] After the fully connected layer outputs, the softmax function is applied to map the feature vectors output in the real domain to a probability distribution. The softmax function transforms any feature vector output from the fully connected layer into a discrete probability distribution. This function ensures that the output satisfies the probability axioms through exponential transformation and normalization operations, given an input vector. (in (total number of categories), the output vector of the softmax function As shown in the following formula:

[0118] ;

[0119] In the formula, For the first The output vector of the softmax function For the first An exponential function of input vectors, For the first An exponential function of input vectors, This is the category number.

[0120] In the FPGA hardware implementation, the calculation of this function involves critical exponential operations, which often require calling resource-intensive CORDIC IP cores. Subsequent division operations also rely on divider IP cores, significantly increasing hardware logic resource overhead and computational latency. Notably, the final classification decision depends solely on the maximum index in the classification probability distribution. Based on this, this application proposes an effective hardware optimization strategy: directly comparing the raw values ​​output by the fully connected layer and obtaining its maximum index to arrive at the classification result (label, the classification label). The data flow diagram is shown in Figure 34. This completely avoids the computationally intensive exponential transformation and division operations in the softmax function, significantly reducing the number of required IP cores and their calling frequency. It also greatly saves critical logic resources in the FPGA, such as lookup tables (LUTs), registers (FFs), and digital signal processing (DSP) blocks, and effectively shortens end-to-end system processing latency, making it particularly suitable for embedded inference scenarios with stringent real-time requirements.

[0121] The simulation output waveform of this module (Figure 35) is compared with the parameters of the forward propagation verification network built in PyCharm (Figure 36) to verify the integrity of the module's function. The data of the two are consistent, which indicates that the module is functioning normally.

[0122] Under a 50MHz system clock, the classification and recognition latency of the entire system was controlled within 2.66ms during the single-board test, as shown in Figure 37, meeting the system's real-time requirements. Meanwhile, on a computer configured with an Intel(R) Core(TM) i5-10400F CPU @ 2.90GHz, 16GB DDR4, and an NVIDIA GeForce RTX 2060, a single classification and recognition operation took 332ms, as shown in Figure 38. This demonstrates the advantages of customized hardware architecture in low-latency edge inference scenarios. This comparison further highlights the technical value of this application in achieving high real-time classification and recognition perception on resource-constrained devices.

[0123] The second embodiment of this application provides a lightweight millimeter-wave radar two-dimensional feature map classification system for FPGA hardware deployment, including:

[0124] The preprocessing module is used to acquire feature maps and perform preprocessing to obtain preprocessed feature maps, construct datasets, and divide the datasets into training sets and validation sets.

[0125] The model building module is used to build a classification network model based on LeNet, and to train the classification network model using the training set to obtain the trained classification network model.

[0126] The validation module is used to verify whether the performance of the trained classification network model meets the performance index using the validation set. If it does, the validated classification network model is obtained and deployed to the FPGA; otherwise, training continues to update the model parameters until the validated performance meets the performance index.

[0127] The classification module uses a classification network model deployed in the FPGA to classify the feature maps and obtain the classification results.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment, characterized in that, Includes the following steps: The feature map is acquired and preprocessed to obtain the preprocessed feature map. A dataset is constructed and the dataset is divided into a training set and a validation set. A classification network model is built based on LeNet. The model is trained using the training set to obtain a trained classification network model. The performance of the trained classification network model is verified using the validation set to determine if it meets the performance metrics. If it does, the validated classification network model is obtained and deployed on an FPGA. The feature map is then classified based on the classification network model to obtain the classification result. Otherwise, training continues to update the parameters of the classification network model until the validated performance meets the performance metrics. The classification network model includes an input layer, a first convolutional layer, a first max-pooling layer, a second convolutional layer, a second max-pooling layer, a channel expansion projection layer with a kernel size of 1×1, a stride of 1, and zero padding, a global max-pooling layer, and a fully connected layer. The preprocessed feature map is input into the input layer. After convolution in the first convolutional layer, a feature map after the first convolution is obtained. This feature map is then input into the first max-pooling layer for pooling to obtain a feature map after the first pooling. The input is processed in the second convolutional layer to perform a convolution operation, resulting in a feature map after the second convolution. The first and second max pooling layers are input into the second max pooling layer for pooling operations to obtain the feature map after the second pooling; the input layer is input into the channel expansion projection layer for linear transformation to obtain the projected feature map; the input layer is input into the global max pooling layer for pooling operations to obtain the feature vector; the input layer is input into the fully connected layer for linear transformation to obtain the classification result; during the deployment of the first and second max pooling layers to the FPGA, a comparison architecture based on dual-row pipelines is set up. When the even-numbered rows of the feature map after the first convolution are input, the maximum value within the window is extracted for adjacent pixels in the same pooling window within the row through a parallel comparator to obtain the local maximum value of the even-numbered row. The local maximum value is obtained by comparing pixel pairs within the odd-numbered rows of the input data stream in real time. Simultaneously, the local maximum value of the even-numbered rows that matches the current odd-numbered row column coordinates is read, and the maximum values ​​of the first and second max-pooling layers are determined through comparison. During the deployment of the channel expansion projection layer and the global max-pooling layer, the channel expansion projection layer and the global max-pooling layer are cascaded. Channel expansion is performed using several convolutional kernels to expand the number of channels in the feature map after the second pooling. The spatial dimension of the feature map after the second pooling is compressed using the global max-pooling layer.

2. The lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in claim 1, characterized in that, During the deployment of the first and second convolutional layers onto the FPGA, a dual-parallel architecture is set up to calculate the product of the preprocessed feature map and the weight parameters. The dual-parallel architecture is based on column-parallel data prefetching, synchronously latching the preprocessed feature map according to the column size of the convolution kernel to complete parallel loading. By deploying a dynamic data rearrangement network, the serially input pixel columns are reconstructed into a spatially parallel pixel array according to the sliding window time sequence. The reconstructed pixel array and the weight parameters are spatially aligned through a multiplication array, and synchronous fixed-point multiplication is performed on the pixels at corresponding positions to obtain the product.

3. The lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in claim 1, characterized in that, In the FPGA, a softmax function with omitting exponential normalization is deployed at the output of the fully connected layer. The maximum value index is selected as the classification result by comparing the numerical values ​​of the output vectors.

4. The lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in claim 1, characterized in that, The validation set is input into the trained classification network model to perform forward propagation, the classification result predicted by the trained classification network model is calculated, and it is compared with the actual classification result to evaluate the generalization ability of the trained classification network model; the forward propagation process is reproduced and verified in the FPGA to determine whether the trained classification network model meets the performance index.

5. The lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in claim 1, characterized in that, The process of extracting the maximum value within a window using the parallel comparator includes: comparing adjacent pixels in the column coordinates of the even-numbered rows of the data stream, and temporarily storing the local maximum value of the even-numbered rows in a FIFO.

6. The lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in claim 1, characterized in that, The preprocessing refers to performing grayscale binarization on the feature map and rotating it counterclockwise by 90° to obtain the preprocessed feature map.

7. A lightweight millimeter-wave radar two-dimensional feature map classification system for FPGA hardware deployment, applied to the lightweight millimeter-wave radar two-dimensional feature map classification method for FPGA hardware deployment as described in any one of claims 1-6, characterized in that, include: The module includes a preprocessing module, a model building module, a validation module, and a classification module. The preprocessing module is used to acquire and preprocess feature maps to obtain preprocessed feature maps, construct a dataset, and divide the dataset into a training set and a validation set. The model building module is used to build a classification network model based on LeNet, and train the classification network model using the training set to obtain a trained classification network model. The validation module is used to verify whether the performance of the trained classification network model meets the performance indicators using the validation set. If yes, the validated classification network model is obtained and deployed to the FPGA; otherwise, training continues to update the parameters of the classification network model until the validated performance is achieved. The performance indicators are met; the classification module uses a classification network model deployed in the FPGA to classify the feature map and obtain the classification result; the classification network model includes an input layer, a first convolutional layer, a first max pooling layer, a second convolutional layer, a second max pooling layer, a channel expansion projection layer with a kernel size of 1×1, a stride of 1, and padding of 0, a global max pooling layer, and a fully connected layer; the preprocessed feature map is input into the input layer, and after convolution in the first convolutional layer, the feature map after the first convolution is obtained; it is then input into the first max pooling layer for pooling to obtain the feature map after the first pooling; The input is processed in the second convolutional layer to perform a convolution operation, resulting in a feature map after the second convolution. The input is processed by the second max pooling layer to obtain the feature map after the second pooling; the input is processed by the channel expansion projection layer to obtain the projected feature map; the input is processed by the global max pooling layer to obtain the feature vector; the input is processed by the fully connected layer to obtain the classification result.

Images