A millimeter wave radar-based target tracking method, system, and device

Abstract

The application provides a target tracking method, system and device based on a millimeter wave radar, the method comprising: extracting local spatial topological features of a radar point cloud by using a graph neural network (GNN), and performing time sequence trajectory deduction by using a long short-term memory network (LSTM) to form a cascaded pure data-driven architecture; wherein the GNN module introduces a spatial attention mechanism to accurately regress the target center coordinates, and the LSTM module automatically enters an autoregressive prediction mode when the target is blocked, and uses historical memory to smooth the output to predict the trajectory. The application makes full use of the sparse characteristics of the radar point cloud, realizes on-demand activation calculation, significantly reduces the computing power and power consumption, can be deployed on a 3-receiving 4-transmitting radio frequency baseband integrated CMOS single chip, meets the strict size, weight and power consumption constraints of a drone, and realizes high-robustness multi-target continuous tracking.

Description

Technical Field

[0001] This application belongs to the field of new-generation information technology technology, and relates to an intelligent sensing and artificial intelligence algorithm, particularly to a target tracking method, system and device based on millimeter-wave radar. Background Technology

[0002] Against the backdrop of rapid development in the low-altitude economy, the safe flight and functional operation of UAVs (Unmanned Aerial Vehicles) heavily rely on high-precision environmental perception capabilities within the near-field airspace of 1m to 50m. Due to the extremely limited payload capacity and battery capacity of UAVs, their onboard perception sensors must meet stringent SWaP (Size, Weight, and Power) constraints. Among current mainstream environmental perception sensors, visual sensors exhibit significantly reduced reliability in low-light or adverse weather conditions (such as rain, snow, and fog); lidar faces bottlenecks such as high cost and susceptibility to environmental interference. In contrast, millimeter-wave radar, with its excellent electromagnetic wave penetration capabilities and all-weather operation, has become an important technological means in the field of dynamic target detection.

[0003] Unlike the dense image pixels output by visual sensors, millimeter-wave radar outputs "radar point clouds." Physically, radar point clouds are highly scattered collections of points in both time and space. This means that only a very small portion of the field of view contains effective target reflection signals, while the vast majority of the area is blank background. Furthermore, target features often exhibit unstructured and discrete distribution characteristics.

[0004] To meet the perception needs of low-altitude UAVs in complex near-field airspace, some radar systems are designed with bandwidths exceeding 600MHz to achieve a high range resolution of 0.25 meters. However, this increased resolution leads to a surge in the amount of point cloud data output by the radar. Although this data remains highly sparse in the overall three-dimensional airspace, it still poses a significant computational challenge for single-chip digital baseband processing units constrained by SWaP.

[0005] Based on the aforementioned hardware, the system needs to run a target tracking algorithm to achieve continuous localization and trajectory prediction for multiple targets. Existing technologies mainly employ two types of solutions, but both have significant drawbacks:

[0006] Firstly, there are tracking algorithms based on traditional statistical filtering (such as Kalman filtering combined with the global nearest neighbor algorithm).

[0007] These methods primarily rely on mathematical kinematic formulas to predict the target's likely location at the next moment. Then, in the newly scanned radar point cloud, the point spatially closest to this predicted location is selected and assigned to the corresponding target. This method only considers the simple "distance" dimension, lacking an understanding of the target's overall shape and motion context. When faced with complex dynamic scenarios such as multiple targets densely converging or targets being briefly obscured leading to instantaneous loss of radar signals, mathematical models based solely on distance are prone to confusion. The system often incorrectly assigns the trajectory of target A to target B (i.e., a change in target identity), or the track breaks directly due to instantaneous observation gaps, resulting in poor anti-jamming stability.

[0008] Secondly, conventional dense deep learning models (such as convolutional neural networks) can be directly introduced.

[0009] To address the high matching error rate of traditional mathematical methods, some cutting-edge research attempts to directly apply complex neural networks (such as convolutional neural networks) used for processing high-resolution images to radar tracking, dividing the entire radar field of view into a dense two-dimensional grid for global scanning and convolutional calculations. However, as mentioned earlier, radar point clouds exhibit extremely high sparsity, with most areas in the field of view consisting of blank backgrounds devoid of targets. Using such dense neural networks would cause the computing unit to waste a significant amount of computational power and time scanning blank areas, generating a massive number of ineffective multiplication and addition operations. This would not only rapidly deplete the limited on-chip memory resources of the integrated chip but also lead to a dramatic increase in system dynamic power consumption and severe heat generation, making practical deployment impossible on resource-constrained low-power single-chip architectures.

[0010] In summary, how to achieve continuous and robust tracking of multiple targets in complex near-field airspace on a domestically produced single-chip radar platform with low power consumption, small size, and lightweight design has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0011] This application provides a target tracking method, system, and device based on millimeter-wave radar to solve the technical problems in the prior art where traditional tracking algorithms based on distance matching are prone to target identity changes and track breaks, and conventional deep learning models suffer from wasted computing power due to the sparsity of radar point clouds and cannot be deployed on low-power single-chip platforms.

[0012] In a first aspect, this application provides a target tracking method based on millimeter-wave radar, comprising: acquiring an original point cloud sequence with three-dimensional coordinates and Doppler information generated by millimeter-wave radar; the original point cloud sequence comprising: three-dimensional coordinates and radial velocity information;

[0013] The original point cloud sequence is preprocessed to obtain a denoised high-quality point cloud sequence;

[0014] For the denoised high-quality point cloud sequence of the current frame, an undirected graph is constructed based on multi-dimensional feature constraints; wherein, a node corresponds to a point in the point cloud sequence, and if the spatial distance between two nodes is less than a first threshold and the absolute value of the difference between the radial velocities corresponding to the two nodes is less than a second threshold, then a connection edge is established between the two nodes.

[0015] Based on a graph neural network, spatial message passing and feature aggregation are performed on the undirected point cloud graph of the current frame. Then, a spatial attention mechanism is used to extract the super nodes of each target from the aggregated node features to obtain the high-precision observation state of the target in the current frame. The high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target.

[0016] A long short-term memory network is used to perform time-series modeling of the high-precision observation state of each tracked target to predict the trajectory distribution of the target;

[0017] When a target is not observed in the current frame, the corresponding Long Short-Term Memory (LSTM) network enters the autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored in the LSM network.

[0018] In this application, spatiotemporal features of radar point clouds are extracted by cascading graph neural networks and long short-term memory networks, and a spatial attention mechanism is introduced to achieve highly robust multi-target continuous tracking and sub-decimeter-level high-precision positioning that is resistant to identity jumps and occlusion.

[0019] In one implementation of the first aspect, preprocessing the original point cloud sequence to obtain a denoised high-quality point cloud sequence includes: for each point in the original point cloud sequence of the current frame, setting a spatial search radius and a minimum neighbor number threshold; counting the number of neighbor nodes of each point cloud within the spatial search radius; if the number of neighbor nodes is less than the minimum neighbor number threshold, then the point cloud is determined to be an isolated clutter point and is removed, thereby obtaining a denoised high-quality point cloud sequence.

[0020] In one implementation of the first aspect, the multidimensional feature constraint is a spatial distance and radial velocity consistency constraint; including: mapping each radar point in the denoised high-quality point cloud sequence to a node in an undirected graph; calculating the Euclidean distance between two radar points corresponding to any two nodes in the undirected graph; calculating the absolute value of the difference in radial velocities between the two radar points; if the Euclidean distance is less than a first threshold and the absolute value of the difference in radial velocities is less than a second threshold, then establishing a connection edge between the two nodes.

[0021] In one implementation of the first aspect, the use of a graph neural network to process the undirected graph to aggregate the feature information of the nodes and obtain the high-precision observation state of the target in the current frame includes: using a multi-layer graph convolutional network to perform message passing on the undirected graph of the current frame, iteratively updating the hidden state of each node, so that nodes belonging to the same physical target tend to be consistent in the feature space; for nodes within each cluster, obtaining the original attention score of each node through a multi-layer perceptron with shared weights; performing Softmax normalization on the original attention scores of all nodes within the point cloud cluster of the same physical target to obtain the spatial attention weight of each node; and using the spatial attention weight to perform a weighted summation of the three-dimensional coordinates and motion features of the nodes to obtain the high-precision observation state of the target in the current frame.

[0022] In one implementation of the first aspect, a Long Short-Term Memory (LSTM) network is used to perform temporal modeling of the high-precision observation state of each tracked target. The prediction of the target's trajectory distribution includes: performing temporal correlation and trajectory prediction based on the high-precision observation state of the current frame using the LSTM network; including: assigning an independent LSTM network unit sequence to each established track, updating the cell state and hidden state through a gating mechanism, and decoding the hidden state to obtain the target's predicted position at the next moment; if the current frame cannot obtain the observation state due to the target being occluded, an autoregressive prediction mode is entered, and a smooth predicted trajectory is forcibly output using the historical cell state memory stored within the LSTM network to obtain the prediction of the target's position in the next frame, thereby generating the predicted trajectory distribution of the target.

[0023] In one implementation of the first aspect, track management is also included, comprising: performing multi-target track lifecycle management on the high-precision observation state and the predicted trajectory of the target according to multi-target track lifecycle management rules, and generating a continuous and smooth three-dimensional motion track for each target; wherein, the lifecycle management includes: association matching, track initiation and track cancellation.

[0024] In one implementation of the first aspect, the association matching includes: calculating the matching cost between the observed states of all targets in the current frame and the predicted states of each long short-term memory network unit; based on the matching cost, using the Hungarian algorithm to perform optimal matching between the observed states and the predicted states; the track initiation includes: establishing a new track when an observation cannot be associated with any existing track in multiple consecutive frames; the track cancellation includes: deleting the track when a track fails to be associated with any observation within a set number of consecutive frames and the confidence of the long short-term memory network autoregressive prediction is lower than a threshold.

[0025] Secondly, this application provides a target tracking system based on millimeter-wave radar, comprising:

[0026] The data acquisition module is used to acquire the raw point cloud sequence with three-dimensional coordinates and Doppler information generated by the millimeter-wave radar; the raw point cloud sequence includes: three-dimensional coordinates and radial velocity information;

[0027] The point cloud preprocessing module is used to preprocess the original point cloud sequence to obtain a high-quality point cloud sequence after denoising.

[0028] An undirected graph topology construction module is used to construct an undirected graph based on multi-dimensional feature constraints for the denoised high-quality point cloud sequence of the current frame; wherein, nodes correspond to points in the point cloud sequence, and if the spatial distance between two nodes is less than a first threshold and the absolute value of the difference between the radial velocities corresponding to the two nodes is less than a second threshold, then a connection edge is established between the two nodes.

[0029] The GNN (Graph Neural Network) spatial feature extraction module is used to perform spatial message passing and feature aggregation on the undirected point cloud graph of the current frame based on the graph neural network, and to extract the super nodes of each target from the aggregated node features using a spatial attention mechanism, so as to obtain the high-precision observation state of the target in the current frame; wherein, the high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target.

[0030] The LSTM (Long Short-Term Memory) temporal trajectory extrapolation module is used to perform temporal modeling of the high-precision observation state of each tracked target using a long short-term memory network, and predict the trajectory distribution of the target. When a target is not observed in the current frame, the corresponding long short-term memory network enters the autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored in the long short-term memory network.

[0031] Thirdly, this application provides an electronic device. The electronic device includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory to cause the electronic device to perform the millimeter-wave radar-based target tracking method as described in any one of the first aspects.

[0032] As described above, the target tracking method, system, and device based on millimeter-wave radar described in this application have the following beneficial effects:

[0033] (1) The target tracking method based on millimeter-wave radar provided in this application abandons the traditional hard allocation mechanism based on distance matching, and uses graph neural network (GNN) to extract the local spatial topological features of the target, and combines long short-term memory network (LSTM) to maintain historical temporal memory. When faced with dense intersection of multiple targets or brief physical obstruction, the system no longer relies solely on the current instantaneous radar echo, but relies on contextual memory to accurately deduce the target trajectory, fundamentally solving the problems of identity jump and track breakage, and realizing highly robust continuous tracking of multiple targets in complex airspace.

[0034] (2) When the target is physically blocked and the current frame cannot match the radar observation data, the LSTM module in this application can automatically enter the "autoregressive prediction mode", directly call the historical cell state memory stored in the LSTM, and force the smooth output of the predicted coordinates of the next moment, thereby effectively overcoming the problem of track breakage caused by instantaneous radar frame loss, and significantly improving the availability of the system in low-altitude flight environment (such as building blockage scenarios).

[0035] (3) In this application, a spatial attention mechanism based on a shared-weight MLP (Multilayer Perceptron) and a Softmax function is introduced into the GNN module, abandoning the traditional global average pooling. By dynamically scoring and weighting the contribution of different reflection points within the target cluster, it is possible to adaptively filter out detached noise and accurately regress the three-dimensional coordinates of the "super node" representing the physical center of the target, thereby further improving the positioning accuracy on the basis of a 0.25-meter distance resolution.

[0036] (4) The system provided in this application makes full use of the highly sparse characteristics of radar point clouds and combines the on-demand activation mechanism of graph neural networks to avoid performing invalid multiplication and addition operations on a large number of blank backgrounds in the field of view. Compared with traditional dense deep learning models, it significantly reduces the amount of computation and on-chip memory usage, and does not require an external high-energy-consuming independent NPU (Neural Processing Unit). At the same time, this system can be effectively deployed on a 3R4T (3R4T) radio frequency baseband integrated CMOS (Complementary Metal-Oxide-Semiconductor) single-chip architecture. While achieving a bandwidth of more than 600MHz and a high distance resolution of 0.25 meters, it maintains extremely low dynamic power consumption and heat generation level, perfectly meeting the strict constraints of UAVs on size, weight and power consumption (SWaP), and has good prospects for engineering implementation and domestic promotion. Attached Figure Description

[0037] Figure 1The diagram shows an application scenario of the hardware structure corresponding to the millimeter-wave radar-based target tracking method described in this application.

[0038] Figure 2 The diagram shown is a schematic representation of the overall process of the target tracking method based on millimeter-wave radar as described in the embodiments of this application.

[0039] Figure 3 The diagram shown is a flowchart of step S2 of the target tracking method based on millimeter-wave radar described in an embodiment of this application.

[0040] Figure 4 The diagram shown is a flowchart of step S3 of the target tracking method based on millimeter-wave radar described in this application embodiment.

[0041] Figure 5 The diagram shown is a flowchart of step S4 of the target tracking method based on millimeter-wave radar described in this application embodiment.

[0042] Figure 6 The diagram shown is a schematic of the feature aggregation process based on graph neural networks and spatial attention mechanism in the target tracking method based on millimeter-wave radar described in this application embodiment.

[0043] Figure 7 The diagram shows the LSTM-based temporal trajectory prediction and anti-occlusion correlation flow of the target tracking method based on millimeter-wave radar described in the embodiments of this application.

[0044] Figure 8 The diagram shown is a schematic representation of the structural composition of the millimeter-wave radar-based target tracking system described in this application embodiment.

[0045] Figure 9 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0046] Component designation explanation

[0047] 1 Target tracking device based on millimeter-wave radar 11 millimeter-wave radar 12 Analog-to-digital converter 13 Digital baseband processing circuit 14 Tracking algorithm processing circuit 21 Data acquisition module 22 Point cloud preprocessing module 23 Undirected graph topology building module 24 GNN spatial feature extraction module 25 LSTM time-series trajectory extrapolation module 26 Track Management Module 30 electronic devices 31 processor 32 memory S1~S6 step Detailed Implementation

[0048] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0049] It should be noted that, in the embodiments of this application, the words "specifically" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "specifically" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "specifically" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0051] To address the problems in the background art described above, this application provides a target tracking method, system, and device based on millimeter-wave radar, aiming to solve the problems of identity switching, track breakage, and the difficulty in deploying conventional deep learning models on low-power single chips in existing multi-target tracking technologies. Meanwhile, to make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit the invention.

[0052] Before providing a further detailed description of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0053] <1> SWaP (Size, Weight, and Power) is a core design constraint in the aerospace and drone industries, referring to the extreme lightweighting of devices in terms of size, weight, and power consumption.

[0054] <2> RF-Baseband Integrated System-on-Chip (SoC): In this embodiment, it is a chip technology used to integrate high-frequency radio frequency transceiver analog circuits and digital signal processing baseband circuits on the same CMOS (Complementary Metal-Oxide-Semiconductor) wafer.

[0055] <3> Range resolution: The radar's ability to distinguish between two targets at very close range, and is inversely proportional to the system's sweep bandwidth. This scheme has a bandwidth greater than 600MHz, and a theoretical resolution of 0.25 meters.

[0056] <4> GNN (Graph Neural Network): A deep learning model specifically designed for processing graph-structured data, adept at extracting local topological relationships between unstructured scattered data.

[0057] <5> LSTM (Long Short-Term Memory): A type of temporal recurrent neural network that excels at processing and predicting important events in time series with relatively long intervals and delays.

[0058] This application provides a target tracking method based on millimeter-wave radar, a target tracking system based on millimeter-wave radar, a target tracking device based on millimeter-wave radar, a storage medium storing an executable program for implementing the target tracking method based on millimeter-wave radar, and an electronic device.

[0059] The following embodiments of this application provide a target tracking method, system, and device based on millimeter-wave radar, including but not limited to hardware application scenarios such as millimeter-wave radar, analog-to-digital converter, chip, on-chip memory, and digital baseband processing circuit. The following description will take a target tracking device based on millimeter-wave radar as an example.

[0060] Please see Figure 1 The diagram shows a schematic representation of the hardware structure and application scenario of the target tracking method based on millimeter-wave radar described in this application.

[0061] like Figure 1 As shown, this embodiment provides a hardware application scenario for a target tracking device based on millimeter-wave radar. The specific hardware structure mainly includes: millimeter-wave radar 11, ADC (Analog-to-Digital Converter) 12, digital baseband processing circuit 13, and tracking algorithm processing circuit 14.

[0062] In one embodiment, the millimeter-wave radar 11 is preferably a three-transmitter, four-receiver single-chip millimeter-wave radar. The three-transmitter, four-receiver single-chip millimeter-wave radar is used to transmit frequency-modulated continuous wave radar signals and receive reflected echo signals from the target.

[0063] The analog-to-digital converter 12 has its input terminal connected to the output terminal of the millimeter-wave radar 11, and is used to convert the analog intermediate frequency signal received by the radio frequency front end into a digital signal for subsequent digital signal processing.

[0064] The digital baseband processing circuit 13, whose input terminal is connected to the output terminal of the analog-to-digital converter 12, is typically integrated into an integrated RF baseband SoC. It is used to perform a series of processing on the digital signal after ADC conversion, including fast Fourier transforms in the distance, velocity, and angle dimensions, as well as constant false alarm rate detection. The final output is a sparse point cloud sequence containing information such as the target's three-dimensional coordinates, radial Doppler velocity, and signal-to-noise ratio.

[0065] The tracking algorithm processing circuit 14 is connected to the digital baseband processing circuit 13 and can be integrated into the dedicated processing core, hardware accelerator or general processor in the above SoC. It is used to run the lightweight spatiotemporal hybrid tracking algorithm proposed in this technology, receive the point cloud sequence output by the digital baseband processing unit, and perform steps such as point cloud denoising, dynamic graph construction, GNN spatial feature aggregation, LSTM temporal prediction and association, and trajectory lifecycle management, and finally output a continuous and stable multi-target motion trajectory.

[0066] It should be noted that the digital baseband processing circuit and the tracking algorithm processing circuit can be integrated into the same RF baseband integrated CMOS chip. The tracking algorithm processing circuit can be composed of dedicated hardware acceleration circuits on the chip (such as an NPU or DSP (Digital Signal Processor) core), or implemented by a general-purpose processor core executing pre-defined program instructions.

[0067] Specifically, the transmitting channel in the three-transmitter, four-receiver single-chip millimeter-wave radar RF front-end radiates frequency-modulated continuous wave radar signals outward through its four transmitting antennas. When the signal encounters an aerial target (such as a drone or bird), some of the energy is reflected. The three receiving antennas of the RF front-end receive these weak reflected echo signals. This process enables active detection of the probe airspace and capture of the original echo signal.

[0068] For example, an RF front-end can be a frequency-modulated continuous wave radar, operating in the 76-81 GHz band, which has a high bandwidth to obtain high range resolution. The RF front-end typically includes circuits such as a voltage-controlled oscillator, power amplifier, transmitting antenna, receiving antenna, low-noise amplifier, and mixer, used to generate the transmitted signal and down-convert the echo signal to an intermediate frequency signal.

[0069] An analog-to-digital converter (ADC) takes the received analog echo signal (such as an intermediate frequency signal) from the radio frequency front-end and outputs it to the ADC. The ADC samples the analog signal at a specific sampling rate and converts it into a discrete digital signal sequence, laying the foundation for subsequent all-digital signal processing.

[0070] Then, the digital baseband processing circuit receives the digital signal output by the ADC and executes a series of standard radar signal processing algorithms, such as:

[0071] (1) Range dimension processing: A fast Fourier transform is performed on each different linear frequency modulated signal chirp for each receiving antenna to convert the signal from the time domain to the frequency domain and obtain the target range information. Thanks to the system bandwidth of more than 600MHz, this step can provide a high range resolution of about 0.25 meters.

[0072] (2) Velocity dimension processing: Perform FFT (Fast Fourier Transform) on the signals of multiple consecutive chirps in the slow time dimension to extract the Doppler frequency shift of the target, thereby calculating the radial velocity of the target.

[0073] (3) Angle dimension processing: Using an array formed by three receiving antennas, the azimuth and elevation angles of the target are estimated by beamforming algorithms such as DBF (Digital Beam Forming) or Capon, and the three-dimensional spatial coordinates are obtained.

[0074] (4) Intelligent tracking processing: Constant false alarm rate detection is performed on the data cube after 3D FFT processing to screen out real target points with energy significantly higher than noise and clutter. Finally, a feature vector containing 3D coordinates, radial velocity, signal-to-noise ratio and other features is generated for each detected target point, and the output is a sparse radar point cloud.

[0075] Finally, the tracking algorithm processing circuit receives the time-series point cloud frames output in real time from the digital baseband processing circuit and executes the core lightweight spatiotemporal hybrid tracking algorithm of this scheme, as follows:

[0076] (1) Point cloud preprocessing: First, density-based filtering is performed on the input raw point cloud to remove isolated noise points and purify the input data.

[0077] (2) Dynamic graph construction: Each point cloud in the current frame is regarded as a node in the graph. The Euclidean distance and velocity difference between nodes are calculated. If two nodes are spatially close enough and move at the same speed, an edge is established between them, thereby transforming the unstructured point cloud into a dynamic graph structure that represents the local spatial relationship of the target.

[0078] (3) Spatial feature aggregation: The constructed dynamic graph is input into the graph neural network. The GNN enables the features of multiple scattering points (nodes) belonging to the same physical target to interact and fuse through multi-layer message passing. Then, the contribution weight of each node to the overall state of the target is calculated through the spatial attention mechanism, and the features of all nodes are weighted and summed to finally output several "super nodes". Each "super node" represents the refined state of a tracked target (such as the position of the centroid, velocity, etc.).

[0079] (4) Temporal trajectory prediction and association: Maintain an LSTM unit for each active trajectory. Input the current frame target observation state (or null value when occluded) output by GNN and the historical state stored in LSTM into the network. LSTM updates its internal memory and predicts the target state at the next time step. Use the Hungarian algorithm to perform optimal matching between the current frame observation and all LSTM predictions to complete data association.

[0080] (5) Anti-occlusion processing: When a target is temporarily occluded and the GNN cannot output its observation state, the corresponding LSTM automatically enters the "autoregressive prediction mode" and extrapolates the target trajectory by relying solely on its powerful temporal memory ability until the target reappears and is re-associated.

[0081] (6) Track management: Based on the matching results, initialize new tracks for new targets and cancel old tracks for targets that are continuously lost.

[0082] It should be noted that those skilled in the art should understand that the specific models, connection methods, scheduling cycles of the embedded processors, and the number of filtered data, etc., of the aforementioned hardware can all be adjusted according to the actual application scenario without departing from the scope of protection of this application.

[0083] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

[0084] In summary, the aforementioned hardware components provide high-resolution raw point cloud data through a high-bandwidth RF front-end and digital baseband. Leveraging a highly integrated chip architecture, they lay a low-power, compact hardware foundation for subsequent algorithm processing. Most importantly, the lightweight algorithm running on the tracking algorithm processing unit fully utilizes the sparsity of point cloud data and the locality of graph computation, avoiding the computational waste associated with traditional intensive computation. This allows complex AI tracking models to run efficiently on a single-chip platform with stringent SWaP constraints, ultimately achieving highly robust and continuous tracking of multiple maneuvering targets in complex near-field airspace.

[0085] The target tracking method based on millimeter-wave radar in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0086] Therefore, this application takes a drone equipped with a 3R4T (3R4T) RF baseband integrated CMOS chip as an example. The system bandwidth is greater than 600MHz and the distance resolution can reach 0.25 meters.

[0087] Please see Figure 2The diagram shows the overall flow chart of the target tracking method based on millimeter-wave radar described in the embodiments of this application.

[0088] like Figure 2 As shown, this embodiment provides a target tracking method based on millimeter-wave radar, the method including the following steps:

[0089] S1, acquire the raw point cloud sequence with three-dimensional coordinates and Doppler information generated by the millimeter-wave radar. The raw point cloud sequence includes, but is not limited to, three-dimensional coordinates and radial velocity information.

[0090] In one embodiment, a radar sensor based on a 3R4T single-chip architecture, i.e., a single-chip radar with three receiving channels and four transmitting channels, is used to control its transmitting array to transmit multiple orthogonal frequency-modulated continuous wave signals sequentially or simultaneously, and the receiving array is used to receive the echo signals reflected from the target scene. For each receiving channel, the received echo signal is mixed with the corresponding transmitting signal to obtain an intermediate frequency signal. After the high-frequency components are filtered out by a low-pass filter, the analog intermediate frequency signal is converted into a digital baseband signal by an analog-to-digital converter, thereby obtaining the original digital echo data containing target distance, speed and angle information.

[0091] Specifically, firstly, a linear frequency modulated continuous wave signal is transmitted through the transmit channel array in the single-chip radar. The transmit channel array adopts a 3R4T configuration, meaning it includes three receive channels and four transmit channels.

[0092] Then, the echo signals reflected from the target scene are simultaneously received through the three receiving channels, and the echo signals from each receiving channel are mixed with the transmitted signals to obtain intermediate frequency signals.

[0093] Next, the intermediate frequency signal is sequentially subjected to analog-to-digital conversion, range-dimensional Fourier transform, and Doppler-dimensional Fourier transform to generate a range-Doppler heatmap. Simultaneously, based on the virtual antenna array formed by the four transmitting channels and three receiving channels, angle estimation is performed on each detection unit to obtain azimuth and elevation angle information.

[0094] Then, constant false alarm rate detection is performed on the distance-Doppler heatmap to extract the target points. Combined with azimuth and pitch information, the coordinate components of each target point in three-dimensional space are calculated to form three-dimensional point cloud data containing X-axis, Y-axis and Z-axis coordinates.

[0095] Next, the Doppler frequency corresponding to each target point is extracted from the Doppler Fourier transform results and converted into a radial velocity value according to the radar operating wavelength, which is then added to the corresponding three-dimensional point cloud data as Doppler information.

[0096] Finally, the point cloud data extracted from consecutive frames, containing 3D coordinates and Doppler information, are organized into a point cloud sequence output in chronological order.

[0097] S2, preprocess the original point cloud sequence to obtain a high-quality point cloud sequence after denoising.

[0098] The original point cloud frame sequence output by the millimeter-wave radar front end after passing through Fast Fourier Transform (FFT) and CFAR (Constant False Alarm Rate Detection) is used as the input data for preprocessing and subsequent processing.

[0099] Please see Figure 3 The diagram shows a flowchart of step S2 of the target tracking method based on millimeter-wave radar described in this embodiment. Figure 3 As shown, the specific steps of S2 include:

[0100] S21, For each point in the original point cloud sequence of the current frame, set the spatial search radius and minimum neighbor number threshold;

[0101] S22, count the number of neighboring nodes of each point cloud within the spatial search radius;

[0102] S23, if the number of neighboring nodes is less than the minimum neighbor number threshold, then the point cloud is determined to be an isolated clutter point and is removed to obtain a high-quality point cloud sequence after denoising.

[0103] In one embodiment, a density filter is used, and a spatial search radius and a minimum neighbor number threshold are set. A neighborhood search is performed on each point cloud. If the number of neighbor nodes of a point within the search radius is less than the minimum neighbor number threshold, the point is determined to be clutter noise and is removed. The denoised high-quality point cloud sequence is output as the input for subsequent steps.

[0104] Specifically, for each point in the current frame point cloud, a spatial search radius and a minimum neighbor number threshold are set; the number of neighbor nodes within the spatial search radius of the point is counted; if the number of neighbor nodes is less than the minimum neighbor number threshold, the point is determined to be an isolated clutter point and is removed; and a high-quality point cloud sequence after denoising is output.

[0105] Specifically, at time The system received a frame containing The effective point cloud set of a reflection point is defined as follows: :

[0106]

[0107] Each point cloud It is a multidimensional feature vector, specifically represented as:

[0108]

[0109] in, This indicates the three-dimensional spatial position of the target point in the Cartesian coordinate system; This indicates the radial Doppler velocity at that point; This represents the signal-to-noise ratio at that point.

[0110] To eliminate the interference of detached noise points on the graph network topology construction, the algorithm first filters out isolated points using a density filter: setting the spatial search radius. and minimum number of neighbors If a certain point In radius The number of neighboring nodes is less than If the noise is detected, it is identified as clutter and removed, and a high-quality point cloud sequence after denoising is output.

[0111] It should be noted that the above content describes the specific implementation process of this step. Those skilled in the art can adjust the above embodiments according to actual radar parameters (such as the number of channels, the number of beams, the beam pointing angle, etc.) without departing from the core idea of ​​the present invention.

[0112] S3. For the denoised high-quality point cloud sequence of the current frame, construct an undirected graph based on multi-dimensional feature constraints. Nodes correspond to points in the point cloud sequence. If the spatial distance between two nodes is less than a first threshold and the absolute value of the difference in radial velocities between the two nodes is less than a second threshold, then a connection edge is established between the two nodes.

[0113] In this process, the multidimensional feature constraint is a consistency constraint between spatial distance and radial velocity.

[0114] Traditional tracking algorithms often employ hard allocation mechanisms based on Euclidean distance, making them ill-suited for scenarios with densely overlapping targets or uneven point cloud distribution. To overcome these limitations, this scheme... Constructing an undirected graph on a point cloud set This provides structured input for message passing in subsequent graph neural networks (GNNs).

[0115] Please see Figure 4 The diagram shows a flowchart of step S3 of the target tracking method based on millimeter-wave radar described in this embodiment. Figure 4 As shown, the specific steps of S3 include:

[0116] S31, map each radar point in the denoised high-quality point cloud sequence as a node of an undirected graph;

[0117] S32, calculate the Euclidean distance between the two radar points corresponding to any two nodes in the undirected graph;

[0118] S33, calculate the absolute value of the difference in radial velocity between the two radar points;

[0119] S34, if the Euclidean distance is less than a first threshold and the absolute value of the difference in radial velocities is less than a second threshold, then a connecting edge is established between the two nodes.

[0120] In one embodiment, in order to more accurately reflect the correlation between scattered points within the same target and suppress false connections between different targets, this scheme does not rely solely on Euclidean distance, but introduces a composite distance metric that integrates spatial location, Doppler velocity, and reflection intensity.

[0121] Specifically, for the point cloud set collected and preprocessed at the current time t, this step constructs an undirected graph.

[0122] First, define the node set. That is, each radar reflection point in each denoised radar point cloud set. Mapping to a graph One of the nodes Each node The carried feature vectors are point clouds Multidimensional features, including its three-dimensional spatial coordinates Radial Doppler velocity and signal-to-noise ratio .

[0123] Then, the edge set is defined. That is, in order to determine whether two radar points belong to the same physical target, this algorithm designs a joint adjacency matrix that integrates "spatial distance" and "velocity consistency". For any two nodes and Calculate its Euclidean distance Doppler velocity difference Furthermore, it can be seen that by introducing distance thresholds and velocity difference thresholds, a connection edge is established between two nodes only when they are physically close and have the same motion trend.

[0124] Furthermore, it can be seen that:

[0125] The elements of the adjacency matrix of the connecting edges are defined as follows:

[0126]

[0127]

[0128]

[0129] in, Represents the Euclidean distance between any two nodes; This represents the Doppler velocity difference between any two nodes; Indicates the distance threshold; Indicates the speed difference threshold; , These represent the coordinates of the three-dimensional geometric center of node i and node j, respectively.

[0130] In summary, the undirected graph constructed using the above method... Its topological structure dynamically reflects the potential object distribution and correlation relationships in the radar point cloud of the current frame. Multiple sparse reflection points belonging to the same physical target will form a connected subgraph or cluster in the graph due to spatial aggregation and similar movement speeds. Points belonging to different targets, or noise points, will not be connected because they are too far apart or have inconsistent movement trends. The graph structure constructed in this way provides an accurate and efficient data foundation for subsequent spatial feature aggregation based on graph neural networks.

[0131] S4. Based on the graph neural network, spatial message passing and feature aggregation are performed on the undirected graph of the point cloud of the current frame, and the super nodes of each target are extracted from the aggregated node features using the spatial attention mechanism to obtain the high-precision observation state of the target in the current frame; wherein, the high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target.

[0132] Please see Figure 5 and Figure 6 The diagrams shown are, respectively, a flowchart of step S4 of the millimeter-wave radar-based target tracking method described in this application embodiment and a flowchart of the feature aggregation process based on graph neural networks and spatial attention mechanisms in the millimeter-wave radar-based target tracking method described in this application embodiment. Figure 5 and Figure 6 As shown, the specific steps of S4 include:

[0133] S41, a multi-layer graph convolutional network is used to perform message passing on the undirected graph of the current frame, and the hidden state of each node is iteratively updated so that nodes belonging to the same physical target tend to be consistent in the feature space.

[0134] S42, For each node within a cluster, the raw attention score of each node is obtained through a multilayer perceptron with shared weights;

[0135] S43, perform Softmax normalization on the original attention scores of all nodes within the same physical target point cloud cluster to obtain the spatial attention weight of each node;

[0136] S44, the spatial attention weights are used to perform a weighted summation of the three-dimensional coordinates and motion features of the node to obtain the high-precision observation state of the target in the current frame.

[0137] In one embodiment, the message passing mechanism of a graph neural network is used to iteratively aggregate and update the features of each node in the graph.

[0138] Specifically, first, after constructing the graph topology, the algorithm uses GNN to perform message passing and extract the topological features of the local scattered point cloud of the same target.

[0139] In this embodiment, a multi-layer graph convolutional network is used, the first... The formula for updating the hidden state of nodes in a layer is:

[0140]

[0141] in, Indicates the first The node feature matrix of the layer; The point cloud features represent the initial input; This represents the adjacency matrix with self-loops added; This represents the corresponding degree matrix; This represents the weight matrix that the network can learn; This represents the ReLU nonlinear activation function.

[0142] Then, after L layers of aggregation, those belonging to the same physical target (cluster) The nodes in the cluster are highly convergent in the feature space. Traditional global average pooling erases the three-dimensional geometric distribution of nodes, leading to a shift in the localization center. To address this, this algorithm introduces a spatial attention mechanism as the feature readout function. The algorithm uses a multilayer perceptron (MLP) with shared weights to evaluate the "contribution" of each node vi in ​​the current cluster to the target physical center and calculates an initial attention score.

[0143] That is, the formula for the original attention score is:

[0144]

[0145] in, Indicates the first The node feature matrix of the layer.

[0146] Next, the scores of all nodes in the cluster are normalized using the Softmax function to obtain the spatial attention weight of each radar point.

[0147] That is, the formula for the spatial attention weight is:

[0148]

[0149] in, This represents the original attention score.

[0150] In other words, this weighting enables the network to automatically suppress low-value ionized noise from propeller edges or multipath reflections (assigning low weights) and focus on high-value, strong-reflection points on the fuselage body (assigning high weights).

[0151] Finally, the attention weights are used to perform a weighted summation of the node's 3D coordinates and motion features to accurately extract the "super-node" representing the target, and output the target's high-precision observation status in the current frame.

[0152] The formula for the high-precision observation state of the target in the current frame is:

[0153]

[0154] in, This indicates the high-precision observation status of the target in the current frame; This represents the high-precision three-dimensional geometric center coordinates of the target obtained from network regression; Indicates the overall speed of motion; Represents a cluster; This represents the spatial attention weight for each radar point.

[0155] In other words, the input in this step is the graph structure representing the correlation of the point cloud in the current frame and the initial feature vectors of each node. The core processing follows the information propagation paradigm of graph neural networks. Each node receives information from its neighboring nodes through the edges of the graph. This application designs an information aggregation strategy to distinguish the importance of the information transmitted by different neighboring nodes and perform weighted fusion accordingly. The aggregated neighborhood information is combined with the node's own features and a new feature representation of the node is generated through an update function containing nonlinear transformation. This process can be stacked in multiple layers in the network, so that the final features of each node can be fused with contextual information from the multi-hop neighborhood.

[0156] Therefore, based on the above principle, the originally isolated point cloud features are embedded in the context of their local graph structure. The output is an updated high-order feature representation for each node, which contains aggregated information about the target's local spatial structure. This makes the point cloud feature representations belonging to the same entity more internally consistent, thus providing a more discriminative feature basis for subsequent cross-frame temporal correlation.

[0157] In summary, this application employs GNN to extract local spatial topology and LSTM to maintain temporal memory. Therefore, when faced with multiple targets closely approaching or brief physical obstruction, the system no longer simply relies on the current instantaneous radar echo to track the target. Instead, it can accurately deduce the target trajectory by relying on contextual memory, fundamentally solving the defects of identity jump and tracking loss, and achieving highly robust continuous tracking of multiple targets.

[0158] S5 employs a long short-term memory network to perform time-series modeling of the high-precision observation state of each tracked target, predicting the target's trajectory distribution.

[0159] Please see Figure 7 The diagram shows the LSTM-based temporal trajectory prediction and anti-occlusion correlation flow of the target tracking method based on millimeter-wave radar described in the embodiments of this application.

[0160] In one embodiment, when a target is not observed in the current frame, the corresponding Long Short-Term Memory (LSTM) network enters an autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored within the LSM network.

[0161] In other words, based on the Long Short-Term Memory (LSTM) network, the high-precision observation state of the current frame is temporally correlated and trajectory predicted; including: assigning an independent LSM network unit sequence to each established track, updating the cell state and hidden state through a gating mechanism, and decoding the hidden state to obtain the target prediction position at the next moment.

[0162] If the current frame cannot obtain the observation state due to the target being occluded, it enters the autoregressive prediction mode, directly using the historical cell state memory stored inside the long short-term memory network to force the output of a smooth prediction trajectory, thereby obtaining the prediction of the target position in the next frame and generating the trajectory distribution of the predicted target.

[0163] Specifically, obtain the current frame High-precision observation status The system then correlates it with historical flight paths in a temporal sequence. In this embodiment, a long short-term memory network is introduced to learn the trajectory distribution of nonlinear maneuvering targets.

[0164] That is: for the first Each track is assigned an independent sequence of LSTM units. At time... Its forward propagation computation process is completed by the following gating mechanism:

[0165] (1) Forget gate and input gate: respectively determine the memory of historical trajectory. The discard ratio and the current observation status. The proportion that is incorporated into memory.

[0166]

[0167]

[0168] in, This indicates the high-precision observation status of the target in the current frame; This indicates the hidden state at the previous moment; The weight matrix representing the forget gate; This represents the weight matrix of the input gate; The bias vector representing the forget gate; This represents the bias vector of the input gate.

[0169] (2) Cell state update and output gate: update internal long-term memory And output the new hidden state. .

[0170]

[0171]

[0172]

[0173]

[0174] in, This indicates the high-precision observation status of the target in the current frame; This represents the internal long-term memory following the renewal of internal cells; This indicates the hidden state of the output at the current moment; A bias vector representing the state of a candidate cell; Indicates the output of the forget gate; Indicates the input gate output; This represents the bias vector of the output gate; This represents the Sigmoid activation function; This represents the weight matrix of the output gate; This indicates the output of the output gate.

[0175] Finally, the hidden state is decoded into the target prediction position at the next time step t+1 using a fully connected layer, i.e.:

[0176]

[0177] in, This represents the bias vector of the prediction layer; This represents the weight matrix of the prediction layer.

[0178] This achieves an occlusion-resistant autoregressive prediction mechanism. When a target is obstructed by buildings while flying at low altitude, the GNN module cannot output the observation state (i.e., When the LSTM algorithm of this application enters the "autoregressive prediction mode", this application directly utilizes the internal cell state. The memories preserved in it make By substituting the data into the network, a smooth and physically consistent predicted trajectory is forced to be output, thus providing a strong resistance to physical occlusion.

[0179] In summary, in this application, when faced with target physical occlusion, i.e. when a certain track fails to match radar observation data in the current frame, the algorithm enters "autoregressive prediction mode", directly calling the historical cell state memory stored inside the LSTM, and forcibly smoothing the output of the predicted coordinates of the next moment, thereby solving the problem of track breakage and identity jump caused by instantaneous radar frame loss.

[0180] It should be noted that in the time series prediction module, in addition to using LSTM, GRU (Gated Recurrent Unit) can also be used to further reduce the number of network parameters; in spatial feature extraction, the lightweight PointNet architecture can also be used to replace GNN for processing unordered point clouds.

[0181] S6. According to the multi-target trajectory lifecycle management rules, perform multi-target trajectory lifecycle management on the high-precision observation state and the predicted trajectory of the target, and generate a continuous and smooth three-dimensional motion trajectory for each target. The lifecycle management includes: association matching, trajectory initiation, and trajectory cancellation.

[0182] In one embodiment, the association matching includes: calculating the matching cost between the observed state of all targets in the current frame and the predicted state of each long short-term memory network unit; and using the Hungarian algorithm to perform optimal matching between the observed state and the predicted state based on the matching cost.

[0183] The track initiation includes: establishing a new track when an observation in multiple consecutive frames cannot be associated with any existing track.

[0184] The track cancellation includes deleting a track when it fails to correlate with any observation within a set number of consecutive frames and the confidence level of the long short-term memory network's autoregressive prediction is below a threshold.

[0185] Specifically, this application establishes a dynamic track management logic matrix, namely:

[0186] (1) Association matching: In order to accurately match the observation state of all current GNN aggregate outputs Compute all current memories in all LSTMs. With all LSTM prediction states The Mahalanobis distance matrix between the two points is used to perform optimal assignment using the Hungarian Algorithm.

[0187] (2) Initiation: If a supernode fails to match any existing track for M consecutive frames, the algorithm initializes a new LSTM unit for it and establishes a new track.

[0188] (3) Track Deletion: If an LSTM track unit is in "autoregressive prediction mode" for N consecutive frames and fails to match actual radar observation data, the system determines that the target has completely left the radar field of view, and then releases the LSTM unit and cancels the track to prevent "ghost tracks".

[0189] Therefore, this application adopts a pure data-driven architecture that cascades "spatial feature extraction based on graph neural networks (GNNs)" and "temporal deduction based on long short-term memory networks (LSTMs)" to process sparse radar point clouds and output continuous multi-target tracks. Simultaneously, this application fully utilizes the highly sparse characteristics of radar point clouds, which resemble "asynchronous event streams," and combines this with the on-demand activation mechanism of graph networks. This effectively avoids the computational waste of traditional neural networks, eliminates the need for an external high-power-consumption independent NPU, and perfectly meets the physical limitations of small size, low power consumption, and low cost of integrated RF baseband CMOS chips.

[0190] In summary, this application employs a Graph Neural Network (GNN) to extract local spatial topological features of the target and combines it with a Long Short-Term Memory (LSTM) network to maintain historical temporal memory. When faced with dense convergence of multiple targets or brief physical occlusion, it can accurately deduce the target trajectory based on contextual memory, fundamentally solving the problems of identity jumps and track breaks, and achieving highly robust continuous tracking of multiple targets. Simultaneously, when the target is occluded and current observation data cannot be obtained, the LSTM module automatically enters an autoregressive prediction mode, smoothly outputting the predicted coordinates for the next moment using internally retained historical cell states, effectively overcoming track breaks caused by instantaneous radar frame drops. Furthermore, this application's method introduces a spatial attention mechanism based on a multilayer perceptron and a Softmax function into the GNN module. By dynamically scoring and weighting the sum of different reflection points within the target cluster, it adaptively filters out detached noise and accurately regresses the three-dimensional coordinates representing the target's physical center, further improving spatial positioning accuracy on top of a high range resolution of 0.25 meters.

[0191] The scope of protection for the target tracking method based on millimeter-wave radar described in this application is not limited to the order of steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0192] This application also provides a target tracking system based on millimeter-wave radar. The target tracking system based on millimeter-wave radar can implement the target tracking method based on millimeter-wave radar described in this application. However, the implementation device of the target tracking method based on millimeter-wave radar described in this application includes, but is not limited to, the structure of the target tracking system based on millimeter-wave radar listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0193] Please see Figure 8 The diagram shows the structural composition of the target tracking system based on millimeter-wave radar as described in the embodiments of this application.

[0194] like Figure 8 As shown, this embodiment provides a target tracking system based on millimeter-wave radar, including: a data acquisition module 21, a point cloud preprocessing module 22, an undirected graph topology construction module 23, a GNN spatial feature extraction module 24, an LSTM temporal trajectory inference module 25, and a track management module 26.

[0195] The data acquisition module 21 is used to acquire the raw point cloud sequence with three-dimensional coordinates and Doppler information generated by the millimeter-wave radar. The raw point cloud sequence includes, but is not limited to, three-dimensional coordinates and radial velocity information.

[0196] In one embodiment, a radar sensor based on a 3R4T single-chip architecture, i.e., a single-chip radar with three receiving channels and four transmitting channels, is used to control its transmitting array to transmit multiple orthogonal frequency-modulated continuous wave signals sequentially or simultaneously, and the receiving array is used to receive the echo signals reflected from the target scene. For each receiving channel, the received echo signal is mixed with the corresponding transmitting signal to obtain an intermediate frequency signal. After the high-frequency components are filtered out by a low-pass filter, the analog intermediate frequency signal is converted into a digital baseband signal by an analog-to-digital converter, thereby obtaining the original digital echo data containing target distance, speed and angle information.

[0197] The point cloud preprocessing module 22 is used to preprocess the original point cloud sequence to obtain a high-quality point cloud sequence after denoising.

[0198] In this embodiment, for each point in the original point cloud sequence of the current frame, a spatial search radius and a minimum neighbor number threshold are set; the number of neighbor nodes of each point cloud within the spatial search radius is counted; if the number of neighbor nodes is less than the minimum neighbor number threshold, the point cloud is determined to be an isolated clutter point and is removed to obtain a high-quality point cloud sequence after denoising.

[0199] The undirected graph topology construction module 23 is used to construct an undirected graph based on multi-dimensional feature constraints from the denoised high-quality point cloud sequence of the current frame. Nodes correspond to points in the point cloud sequence. If the spatial distance between two nodes is less than a first threshold and the absolute value of the difference in radial velocities between the two nodes is less than a second threshold, then a connection edge is established between the two nodes.

[0200] In this embodiment, each radar point in the denoised high-quality point cloud sequence is mapped to a node in an undirected graph; the Euclidean distance between the two radar points corresponding to any two nodes in the undirected graph is calculated; the absolute value of the difference in radial velocity between the two radar points is calculated; if the Euclidean distance is less than a first threshold and the absolute value of the difference in radial velocity is less than a second threshold, then a connection edge is established between the two nodes.

[0201] The GNN spatial feature extraction module 24 is used to perform spatial message passing and feature aggregation on the undirected point cloud graph of the current frame based on the graph neural network, and to extract the super nodes of each target from the aggregated node features using a spatial attention mechanism, thereby obtaining the high-precision observation state of the target in the current frame. The high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target.

[0202] In this embodiment, a multi-layer graph convolutional network is used to perform message passing on the undirected graph of the current frame, iteratively updating the hidden state of each node, so that nodes belonging to the same physical target tend to be consistent in the feature space; for each node in each cluster, the original attention score of each node is obtained through a multi-layer perceptron with shared weights; the original attention scores of all nodes in the point cloud cluster of the same physical target are subjected to Softmax normalization to obtain the spatial attention weight of each node; the spatial attention weight is used to perform weighted summation of the three-dimensional coordinates and motion features of the node to obtain the high-precision observation state of the target in the current frame.

[0203] The LSTM time-series trajectory extrapolation module 25 is used to perform time-series modeling of the high-precision observation state of each tracked target using a long short-term memory network, and predict the trajectory distribution of the target; when a target is not observed in the current frame, the corresponding long short-term memory network enters the autoregressive prediction mode, and outputs the predicted trajectory of the target based on the historical state information stored inside the long short-term memory network.

[0204] In this embodiment, when a target is not observed in the current frame, the corresponding Long Short-Term Memory (LSTM) network enters an autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored within the LSM network.

[0205] Specifically, based on the Long Short-Term Memory (LSTM) network, the high-precision observation state of the current frame is temporally correlated and trajectory predicted; including: assigning an independent LSM network unit sequence to each established track, updating the cell state and hidden state through a gating mechanism, and decoding the hidden state to obtain the target prediction position at the next moment.

[0206] If the current frame cannot obtain the observation state due to the target being occluded, it enters the autoregressive prediction mode, directly using the historical cell state memory stored inside the long short-term memory network to force the output of a smooth prediction trajectory, thereby obtaining the prediction of the target position in the next frame and generating the trajectory distribution of the predicted target.

[0207] The track management module 26 is used to perform association matching, track initiation and cancellation, and output continuous multi-target tracks.

[0208] In this embodiment, based on the multi-target trajectory lifecycle management rules, lifecycle management of the high-precision observation state and the predicted trajectory of the target is performed to generate a continuous and smooth three-dimensional motion trajectory for each target. The lifecycle management includes: association matching, trajectory initiation, and trajectory cancellation.

[0209] The structure and principle of the data acquisition module 21, point cloud preprocessing module 22, undirected graph topology construction module 23, GNN spatial feature extraction module 24, LSTM temporal trajectory inference module 25 and track management module 26 correspond one-to-one with the steps in the above target tracking method based on millimeter-wave radar, so they will not be described in detail here.

[0210] In summary, the system provided in this application fully utilizes the highly sparse characteristics of radar point clouds and combines the on-demand activation mechanism of graph neural networks to avoid performing invalid multiplication and addition operations on a large number of blank backgrounds in the field of view. Compared with traditional dense deep learning models, it significantly reduces the amount of computation and on-chip memory usage, and does not require an external high-power independent NPU. At the same time, this system can be effectively deployed on a 3R4T RF baseband integrated CMOS single-chip architecture, achieving a bandwidth of more than 600MHz and a high range resolution of 0.25 meters while maintaining extremely low dynamic power consumption and heat generation levels, perfectly meeting the stringent constraints of UAVs on size, weight and power consumption (SWaP).

[0211] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0212] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0213] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] Please see Figure 9 The diagram shows a structural schematic of the electronic device described in an embodiment of this application. Figure 9 As shown, this embodiment provides an electronic device, the electronic device 30 including a memory 32 and a processor 31.

[0215] The memory 32 is used to store computer programs; preferably, the memory 32 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.

[0216] Specifically, memory 32 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 30 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 32 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application. It is understood that memory 32 may be volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable categories of memory.

[0217] The processor 31 is connected to the memory 32 and is used to execute the computer program stored in the memory 32 so that the electronic device 30 performs the target tracking method based on millimeter-wave radar as described in any embodiment of this application.

[0218] Optionally, the processor 31 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0219] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0220] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0221] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0222] In summary, the target tracking method, system, and device based on millimeter-wave radar provided in this application have the following beneficial effects:

[0223] This application provides a target tracking method, system, and device based on millimeter-wave radar. It abandons the traditional hard allocation mechanism based on distance matching and adopts a graph neural network to extract local spatial topological features. Combined with a long short-term memory network to maintain historical temporal memory, it accurately infers the trajectory by relying on contextual memory in scenarios with dense intersection of multiple targets or physical occlusion. This fundamentally solves the problems of identity jump and track breakage, and achieves highly robust continuous tracking. At the same time, by introducing a spatial attention mechanism to dynamically weight the reflection points within the target cluster, it adaptively filters out noise and accurately regresses the three-dimensional coordinates of the target's physical center, further improving the positioning accuracy on the basis of a high range resolution of 0.25 meters. At the system level, this solution fully utilizes the highly sparse characteristics of radar point clouds and combines them with the on-demand activation mechanism of graph networks to avoid ineffective multiplication and addition operations on a large number of blank backgrounds in the field of view, significantly reducing computing power and on-chip memory usage, and eliminating the need for an external high-power-consuming independent NPU. It can also be effectively deployed on a 3-receive, 4-transmit RF baseband integrated CMOS single-chip architecture, maintaining extremely low dynamic power consumption and heat generation levels while perfectly meeting the stringent constraints of UAVs on size, weight, and power consumption, demonstrating excellent prospects for engineering implementation and domestic promotion. Furthermore, this application has broad applicability. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0224] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0225] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A target tracking method based on millimeter-wave radar, characterized in that, The method includes: Acquire a raw point cloud sequence with three-dimensional coordinates and Doppler information generated by a millimeter-wave radar; the raw point cloud sequence includes: three-dimensional coordinates and radial velocity information; The original point cloud sequence is preprocessed to obtain a denoised high-quality point cloud sequence; For the denoised high-quality point cloud sequence of the current frame, an undirected graph is constructed based on multi-dimensional feature constraints; wherein, a node corresponds to a point in the point cloud sequence, and if the spatial distance between two nodes is less than a first threshold and the absolute value of the difference between the radial velocities corresponding to the two nodes is less than a second threshold, then a connection edge is established between the two nodes. Based on a graph neural network, spatial message passing and feature aggregation are performed on the undirected point cloud graph of the current frame. Then, a spatial attention mechanism is used to extract the super nodes of each target from the aggregated node features to obtain the high-precision observation state of the target in the current frame. The high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target. A long short-term memory network is used to perform time-series modeling of the high-precision observation state of each tracked target to predict the trajectory distribution of the target; When a target is not observed in the current frame, the corresponding Long Short-Term Memory (LSTM) network enters the autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored in the LSM network.

2. The target tracking method based on millimeter-wave radar according to claim 1, characterized in that, Preprocessing the original point cloud sequence to obtain a denoised, high-quality point cloud sequence includes: For each point in the original point cloud sequence of the current frame, set the spatial search radius and minimum neighbor number threshold; Count the number of neighboring nodes of each point cloud within the spatial search radius; If the number of neighboring nodes is less than the minimum neighbor number threshold, the point cloud is determined to be an isolated clutter point and is removed to obtain a high-quality point cloud sequence after denoising.

3. The target tracking method based on millimeter-wave radar according to claim 1, characterized in that, The multidimensional feature constraint is a consistency constraint between spatial distance and radial velocity; including: Each radar point in the denoised high-quality point cloud sequence is mapped to a node of an undirected graph. Calculate the Euclidean distance between the two radar points corresponding to any two nodes in the undirected graph; Calculate the absolute value of the difference in radial velocity between the two radar points; If the Euclidean distance is less than a first threshold and the absolute value of the difference in radial velocities is less than a second threshold, then a connecting edge is established between the two nodes. The formula for the connecting edge is: in, Represents the Euclidean distance between any two nodes; This represents the Doppler velocity difference between any two nodes; Indicates the distance threshold; Indicates the speed difference threshold; , These represent the coordinates of the three-dimensional geometric center of node i and node j, respectively.

4. The target tracking method based on millimeter-wave radar according to claim 1, characterized in that, The undirected graph is processed using a graph neural network to aggregate the feature information of the nodes, resulting in a high-precision observation state of the target in the current frame, including: A multi-layer graph convolutional network is used to perform message passing on the undirected graph of the current frame and iteratively update the hidden state of each node, so that nodes belonging to the same physical target tend to be consistent in the feature space. For each node within a cluster, the raw attention score of each node is obtained through a multilayer perceptron with shared weights; The original attention scores of all nodes within the same physical target point cloud cluster are subjected to Softmax normalization to obtain the spatial attention weight of each node. The spatial attention weights are used to perform a weighted summation of the node's three-dimensional coordinates and motion features to obtain the target's high-precision observation status in the current frame.

5. The target tracking method based on millimeter-wave radar according to claim 4, characterized in that, The formula for the high-precision observation state is: in, This indicates the high-precision observation status of the target in the current frame; This represents the high-precision three-dimensional geometric center coordinates of the target obtained from network regression; Indicates the overall speed of motion; Represents a cluster; This represents the spatial attention weight for each radar point; This represents the raw attention score; Indicates the first The node feature matrix of the layer; The point cloud features represent the initial input; This represents the adjacency matrix with self-loops added; This represents the corresponding degree matrix; This represents the weight matrix that the network can learn; This represents the ReLU nonlinear activation function.

6. The target tracking method based on millimeter-wave radar according to claim 1, characterized in that, Using a long short-term memory network, the high-precision observation state of each tracked target is time-series modeled to predict the target's trajectory distribution, including: Based on the Long Short-Term Memory network, the high-precision observation state of the current frame is temporally correlated and trajectory predicted; including: assigning an independent Long Short-Term Memory network unit sequence to each established track, updating the cell state and hidden state through a gating mechanism, and decoding the hidden state to obtain the target prediction position at the next moment; If the current frame cannot obtain the observation state due to the target being occluded, it enters the autoregressive prediction mode, directly using the historical cell state memory stored inside the long short-term memory network to force the output of a smooth prediction trajectory, thereby obtaining the prediction of the target position in the next frame and generating the trajectory distribution of the predicted target.

7. The target tracking method based on millimeter-wave radar according to claim 1, characterized in that, It also includes track management, including: According to the multi-target trajectory lifecycle management rules, the high-precision observation state and the predicted trajectory of the target are subjected to multi-target trajectory lifecycle management to generate a continuous and smooth three-dimensional motion trajectory for each target. The lifecycle management includes: association matching, track initiation, and track cancellation.

8. The target tracking method based on millimeter-wave radar according to claim 7, characterized in that, The association matching includes: calculating the matching cost between the observed state of all targets in the current frame and the predicted state of each long short-term memory network unit; and using the Hungarian algorithm to perform optimal matching between the observed state and the predicted state based on the matching cost. The track initiation includes: establishing a new track when an observation in multiple consecutive frames cannot be associated with any existing track; The track cancellation includes deleting a track when it fails to correlate with any observation within a set number of consecutive frames and the confidence level of the long short-term memory network's autoregressive prediction is below a threshold.

9. A target tracking system based on millimeter-wave radar, characterized in that, The system includes: The data acquisition module is used to acquire the raw point cloud sequence with three-dimensional coordinates and Doppler information generated by the millimeter-wave radar; the raw point cloud sequence includes: three-dimensional coordinates and radial velocity information; The point cloud preprocessing module is used to preprocess the original point cloud sequence to obtain a high-quality point cloud sequence after denoising. An undirected graph topology construction module is used to construct an undirected graph based on multi-dimensional feature constraints for the denoised high-quality point cloud sequence of the current frame; wherein, nodes correspond to points in the point cloud sequence, and if the spatial distance between two nodes is less than a first threshold and the absolute value of the difference between the radial velocities corresponding to the two nodes is less than a second threshold, then a connection edge is established between the two nodes. The GNN spatial feature extraction module is used to perform spatial message passing and feature aggregation on the undirected point cloud graph of the current frame based on the graph neural network, and to extract the super nodes of each target from the aggregated node features using the spatial attention mechanism, so as to obtain the high-precision observation state of the target in the current frame; wherein, the high-precision observation state includes at least the three-dimensional center coordinates and velocity of the target. The LSTM temporal trajectory extrapolation module is used to perform temporal modeling of the high-precision observation state of each tracked target using a long short-term memory network, and predict the trajectory distribution of the target. When a target is not observed in the current frame, the corresponding long short-term memory network enters the autoregressive prediction mode and outputs the predicted trajectory of the target based on the historical state information stored in the long short-term memory network.

10. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is connected to the memory and is used to execute the computer program stored in the memory to enable the electronic device to perform the target tracking method based on millimeter-wave radar as described in any one of claims 1 to 8.

Images