Dynamic and static target continuous positioning and tracking method and system based on millimeter waves

Abstract

The invention discloses a dynamic and static target continuous positioning and tracking method and system based on millimeter waves. The method comprises the steps that dynamic point cloud data and static point cloud data reflected by a target are acquired through a millimeter wave radar sensor, the dynamic point cloud data reflect target motion state information, and the static point cloud data reflect vital sign information of the target in a static state; fusing the dynamic point cloud data and the static point cloud data to obtain fused point cloud data, and clustering the fused point cloud data by using a density-based clustering algorithm by taking a reflection point for distinguishing a target from an environment as a target to obtain a clustering result; and on the basis of the clustering result, executing continuous positioning and tracking on the plurality of targets to obtain a tracking result. The method can effectively extract the dynamic and static information of the target, achieves the continuous positioning and precise tracking of multiple targets, and is higher in practicality and application value.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things, and more specifically, to a method and system for continuous positioning and tracking of dynamic and static targets based on millimeter waves. Background Art

[0002] Human body tracking and positioning are ubiquitous human perception tasks. By obtaining the position and trajectory of the human body, various applications (such as gait recognition, virtual / augmented reality, etc.) can be developed. In addition, tracking and positioning are the basis for further human perception, including motion recognition, health monitoring, etc. The core task of human body tracking and positioning is how to distinguish the human body from surrounding static objects. Since the reflection signal of the human body is easily confused with other objects, it is necessary to mine specific human-related features to distinguish humans. Moreover, when multi-person positioning is required, continuous tracking is also a basic problem.

[0003] At present, human detection is mainly done by mining the dynamics of the human body. For example, focusing on the dynamics of human motion and using signal variance to detect humans, observing the dynamics of vital signs to achieve human recognition, or using human contours to achieve human detection. For multi-person tracking and localization, researchers usually regard it as an association problem between detection results and existing trajectories. It can be transformed into a bipartite graph matching problem and can be well solved by some classic algorithms (such as the Hungarian algorithm).

[0004] In the existing technology, mmSense uses a single 60GHz millimeter wave radio frequency to implement a multi-person detection and identification system. mmSense first divides the detection area into multiple areas and constructs the received signal strength fingerprint when there are people and when there are no people. Then, a human detection model based on long short-term memory (LSTM) is used to determine whether there are people in each area.

[0005] mmTrack uses spatial spectrum instead of RSS fingerprint to achieve multi-person positioning. mmTrack is implemented on a 60GHz 802.11ad chipset and is equipped with an additional antenna array. It first extracts channel impulse response (CIR) measurements from the received signal and uses a beamforming algorithm to obtain the spatial spectrum. Taking into account the dynamics of human motion, mmTrack calculates the change in the spatial spectrum energy distribution at each distance to detect the target. After converting the points in the spherical coordinate system to 3D positions in Euclidean space, mmTrack clusters these points using the K-means clustering algorithm to estimate the human body position. mmTrack further exploits the geometric properties of the detected targets and designs post-verification techniques for clustering robustness improvement. Finally, mmTrack models continuous human tracking as a weighted bipartite graph matching problem to perform association between detection results and existing trajectories.

[0006] The spatial distribution of the point cloud shows the outline of the human body and can also be used for human detection. mID uses the sparse point cloud generated by the millimeter wave radar to demonstrate the feasibility of human tracking and identification using millimeter wave radar. This method uses sparse point clouds to track and identify multiple people. The sparse point cloud shows the three-dimensional location of the reflection point and indicates the information of the person. mID first merges the point cloud into clusters using the density-based spatial clustering of applications with noise (DBSCAN). Considering that the distribution of the person point cloud on the z-axis is more scattered than on the x-axis and y-axis, mID modifies the Euclidean distance and reduces the weight of the contribution to the z-axis. Then, mID uses the Hungarian algorithm to associate each cluster at the current timestamp with the previously acquired trajectory. mID further uses the Kalman filter to correct tracking errors. Finally, the points of potential human objects are voxelized by a fixed-size bounding box to form an occupancy grid.

[0007] When the distance between targets is too close for the millimeter wave radar to distinguish, the track crossing problem will occur and reduce the tracking accuracy. To solve this problem, PALMAR applies Adaptive Order Hidden Markov Model (AO-HMM) and Crossover Path Disambiguation Algorithm (CPDA) to deal with multi-person path ambiguity and track crossing.

[0008] After analysis, the existing technology mainly has the following defects:

[0009] 1) Traditional methods often have difficulty effectively distinguishing between static targets and their surroundings, resulting in insufficient feature information to support continuous positioning of the target. Static targets usually stay within the observation range of millimeter-wave radar for a long time, and traditional methods cannot accurately extract the target's tiny motion features, resulting in a decrease in the accuracy of continuous positioning.

[0010] 2) Existing technologies are usually limited by factors such as resolution and signal-to-noise ratio, making it difficult to extract sufficient feature information to accurately describe static targets. In sensors such as millimeter-wave radar, since the reflected signal is weak when the target is in a static state, it may be affected by environmental noise and interference, making the extracted feature information insufficient to support continuous tracking and positioning of static targets.

[0011] 3) Existing methods mainly rely on mining the dynamic features of the human body for human detection. This method can effectively detect and track moving humans to a certain extent. However, when the target is in a static state, this method may cause the point cloud information to become sparse, thus affecting the continuous tracking of the target position. The sparse point cloud information of static targets will bring difficulties to the tracking system due to the lack of sufficient information to accurately determine the target position. In addition, the continuous positioning of static targets is also affected due to the lack of dynamic features to support continuous tracking. Summary of the invention

[0012] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and system for continuous positioning and tracking of dynamic and static targets based on millimeter waves to solve the problem of continuous positioning of static targets encountered by millimeter wave radar in multi-target tracking.

[0013] According to a first aspect of the present invention, a method for continuous positioning and tracking of dynamic and static targets based on millimeter waves is provided. The method comprises the following steps:

[0014] Using a millimeter wave radar sensor to obtain dynamic point cloud data and static point cloud data reflected by the target, the dynamic point cloud data reflects the target's motion state information, and the static point cloud data reflects the target's vital sign information in a static state;

[0015] The dynamic point cloud data and the static point cloud data are fused to obtain fused point cloud data, and the fused point cloud data are clustered using a density-based clustering algorithm with the goal of distinguishing reflection points between a target and an environment to obtain a clustering result;

[0016] Based on the clustering results, continuous positioning and tracking of multiple targets are performed to obtain tracking results.

[0017] According to a second aspect of the present invention, a millimeter wave-based continuous positioning and tracking system for dynamic and static targets is provided. The system comprises:

[0018] Point cloud data acquisition module: used to use the millimeter wave radar sensor to acquire dynamic point cloud data and static point cloud data reflected by the target, wherein the dynamic point cloud data reflects the target motion state information, and the static point cloud data reflects the target's vital signs information in a static state;

[0019] Point cloud fusion clustering module: used to fuse the dynamic point cloud data with the static point cloud data to obtain fused point cloud data, and to cluster the fused point cloud data using a density-based clustering algorithm with the reflection points that distinguish the target from the environment to obtain a clustering result;

[0020] Target tracking module: used to perform continuous positioning and tracking of multiple targets based on the clustering results to obtain tracking results.

[0021] Compared with the prior art, the advantages of the present invention are that the millimeter-wave-based method and system for continuous positioning and tracking of dynamic and static targets can effectively extract rich point cloud information from static targets, thereby effectively distinguishing the difference between static targets and the surrounding environment, and realizing continuous tracking and positioning of targets. The present invention can improve the performance of millimeter-wave radar in multi-target scenarios, improve the accuracy and stability of target tracking, and thus expand the scope of application of millimeter-wave radar in practical applications.

[0022] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0024] Figure 1 is a flow chart of a method for continuous positioning and tracking of dynamic and static targets based on millimeter waves according to an embodiment of the present invention;

[0025] Figure 2 is a diagram of an IWR1843BOOST virtual antenna array according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0027] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0028] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0029] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0030] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0031] The present invention proposes a method and system for continuous positioning and tracking of dynamic and static targets based on millimeter waves, which can effectively extract dynamic and static information of targets and realize continuous positioning and tracking of multiple targets, and has high practicality and application value. Figure 1 As shown, the provided method for continuous positioning and tracking of dynamic and static targets based on millimeter waves includes the following steps:

[0032] Step S110: using a millimeter wave radar sensor to obtain dynamic point cloud data reflected by the target, where the dynamic point cloud data reflects the target motion state information.

[0033] In one embodiment, dynamic point cloud data of target reflection is generated according to the following steps:

[0034] Step S111, obtaining the original millimeter wave signal.

[0035] For a target at a distance d from the radar, the IF (Intermediate Frequency) signal generated by the mixer of the transmit and receive signals of each Tx-Rx pair can be simply defined as:

[0036]

[0037] Where α represents the amplitude of the signal. The frequency and phase of the IF signal are and φ 0 =2πf 0 τ, where τ is the round trip delay, f 0 is the starting frequency of the chirp, and c is the speed of light. is the effective sweep bandwidth B and chirp duration T c The calculated frequency slope, finally N a Tx-Rx pairs can obtain N s ×N c ×N aThe IF complex real signal matrix M, where N s Indicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame. For example, N s =128, N c =128, N a =12.

[0038] Step S112, generating a range-Doppler heat map.

[0039] Generating the range-Doppler heat map includes the processes of range-dimensional Fourier transform, static clutter elimination and Doppler-dimensional Fourier transform of the intermediate frequency signal.

[0040] 1) Distance Dimensional Fourier Transform

[0041] For a single target, the frequency f of the IF signal remains unchanged, and the distance to the target can be calculated as In the case of multiple reflection sources, the signal matrix M is first subjected to windowed FFT (i.e., Range-FFT, distance-dimensional Fourier transform) in the first dimension to obtain a value of Matrix M R ,in is the length of Range-FFT. For example, if the window function is Hanning window,

[0042] 2) Static clutter elimination

[0043] Static clutter elimination is an important step in radar signal processing, which aims to remove the clutter generated by the reflection of stationary objects from the received signal in order to detect dynamic targets more accurately. Specifically, the matrix M R Averaging in the second dimension gives the static clutter matrix The matrix M R and the static clutter matrix After subtraction, the Range-FFT matrix after removing static clutter is obtained

[0044] 3) Doppler Fourier transform

[0045] Assume that the radar c By sending a chirp wave every second and observing the phase difference ω between adjacent chirps, the speed can be estimated Specifically, for the matrix After applying windowed FFT (i.e., Doppler-FFT, Doppler-dimensional Fourier transform) in the second dimension, multiple reflection sources moving at different speeds can be identified by performing Doppler-FFT on the chirp phase, and finally a magnitude of The distance-Doppler-antenna matrix M RDA ,in is the length of Doppler-FFT. RDA After summing in the third dimension, we get the magnitude Distance-Doppler heat map (RD map) For example, the window function is the Hanning window,

[0046] Step S113, detecting high energy points in the heat map based on a set threshold.

[0047] After obtaining the range-Doppler heat map, a suitable energy threshold can be set to detect high-energy points, which usually correspond to targets or regions of interest. Through threshold detection, clutter and background noise in the heat map can be filtered out, retaining the information of the target or region of interest. Specifically, according to the application requirements and the background noise level, a suitable energy threshold θ is set, for example, Usually, energy points above the threshold are considered as candidate points of the target or region of interest. Traverse each pixel in the heat map and compare the energy value of the pixel with the set threshold. If the energy value of the pixel is higher than the threshold, it is marked as a high energy point p i =(r, v), represents the i-th high energy point, the distance of the point is r, and the Doppler velocity is v. Output the pixel coordinates of all the pixels marked as high energy points as the point set P d ={p 1 , p 2 , ..., p n}, where n is the total number of high energy points.

[0048] Step S114, calculating the point cloud angle information.

[0049] Calculating the point cloud angle information includes millimeter wave radar angle estimation modeling, angle Fourier transform and other processes.

[0050] 1) Millimeter-wave radar angle estimation modeling

[0051] By using multiple antennas simultaneously and comparing the phase difference between adjacent receiving antennas, the angle of arrival (AoA) of the target can be estimated. Due to the spatial position differences between the receivers, the signal received by each receiver will have a slight phase difference, which depends on the relative position and AoA of the receiving antennas. If there are multiple antennas in each direction, the azimuth and elevation angles can be calculated, expressed as θ a and θ e , or θ when referring to these two angles (a，e) .

[0052] Assume that there are N in the azimuth and elevation directions a ×N e linearly spaced receiving antennas in different directions θ (a，e)m If there are M targets on the network, each target can be regarded as a signal source, and the receiving antenna array will receive a set of weighted sum signals, represented by x:

[0053]

[0054] Among them, M represents M data sources, s(θ (a，e)m ) is the signal arrival angle θ (a，e)m The steering vector of the phase difference between the receivers, α m represents the amplitude, and n is the noise. AoA estimation can be modeled as estimating θ for each object m based on a set of receive antenna data x. (a,e) value.

[0055] 2) Angle Fourier transform

[0056] Estimate s(θ) of object m in equation (2) (a，e)m ) is to use the correlation between the receiver data x and the steering vector of the candidate angle. This process is equivalent to applying FFT (Angle-FFT, angle Fourier transform) on the receiver data x, because the steering vector can be considered as a set of FFT coefficients that give the information about Δφ. a and Δφ e The frequency components of a represents the azimuth, Δφ e Indicates the pitch angle. For example, using the IWR1843BOOST millimeter-wave radar of Texas Instruments, which has a three-transmit and four-receive antenna array, the virtual antenna layout is as follows Figure 2 As shown, the interval between adjacent antennas is half a wavelength (i.e., λ / 2). The azimuth receiver will form a signal of is the initial phase of the first receiving antenna. The elevation angle receiver will form a signal of Where n is the receive antenna index in each direction.

[0057] By performing azimuth-FFT on the azimuth receiving antennas (Tx1-Rx and Tx2-Rx), Δφ can be obtained. a This will give the frequency Δφ a and initial phase By performing FFT (Elevation-FFT) on the elevation receiver, Δφ can be obtained. a This will give the value of the frequency Δφ a and Phase Therefore, given Δφ a and We can also calculate Δφ e Finally, through the point set P d Calculate Angle-FFT for each point in the d The azimuth and elevation information corresponding to each point in the point set P d The information of the point in is updated to p i =(r, Δφ a , Δφ e , v), at this time point set P d It is the dynamic point cloud reflected by the target.

[0058] Step S120, using a millimeter wave radar sensor to obtain static point cloud data reflected by the target, where the static point cloud data reflects the weak motion or vital sign information of the target in a stationary state.

[0059] In one embodiment, static point cloud data of target reflection is generated according to the following steps:

[0060] Step S121, continuous frame accumulation.

[0061] In order to distinguish static targets from static clutter, an intuitive idea is that even if a person remains still, the body still produces weak vital signs (such as breathing) or small movements, which can be used to locate static targets. Specifically, after performing static clutter removal, the static clutter (i.e., matrix ). By continuously collecting multiple frames of static clutter and superimposing these data frames one by one, we can obtain a The matrix M sc , where N f is the number of data frames. For example, N f =64.

[0062] Step S122, inter-frame static clutter elimination.

[0063] The purpose of inter-frame static clutter elimination is to remove static clutter from continuous frame radar data to improve the radar system's detection performance for static targets with micro-motion. By eliminating static clutter, background interference can be reduced and the characteristics of static targets can be highlighted, so that the radar system can detect and track targets more accurately and improve the performance and reliability of the system. Specifically, the matrix M sc Averaging in the first dimension yields the static clutter matrix The matrix M sc and the static clutter matrix After subtraction, the matrix after removing static clutter is obtained

[0064] Step S123, micro-Doppler FFT.

[0065] Pair Matrix After applying the windowed FFT, i.e., Micro-Motion Doppler Fourier Frequency Transform (MMDoppler-FFT), on the first dimension, we can finally get a value of The matrix M RMMD ,in is the length of MMDoppler-FFT. RMMD After summing in the third dimension, we get the magnitude Distance-Micro Motion Doppler Heat Map For example, the window function is the Hanning window, Based on micro-Doppler FFT, static targets and static objects can be effectively distinguished.

[0066] Step S124, estimating point cloud information

[0067] Similar to the content of step S114, the static point set P can be finally obtained. c , where each point p in the set i =(r, Δφ a , Δφ e , v).

[0068] Step S130 , fusing the dynamic point cloud data with the static point cloud data, and clustering the fused point cloud data to distinguish the reflection points of the target and the environment.

[0069] Specifically, the fusion clustering of dynamic point cloud data and static point cloud data includes the following steps:

[0070] Step S131: perform dynamic and static point cloud clustering to obtain clustering results

[0071] After completing the above steps, two different methods can be used to obtain point clouds from two different "perspectives". These perspectives provide a comprehensive perspective of the target (i.e., static and dynamic). Through point cloud merging, point clouds from different perspectives can be merged together to obtain more comprehensive target point cloud information. In order to reconstruct a denser point cloud reflected by a human target, these point clouds are merged together. First, in order to remove the influence of noise (such as ghosts caused by multipath effects that may obscure the true reflection points from the target). Use the DBSCAN (density-based clustering algorithm) algorithm to cluster different objects or targets. Specifically, first merge the dynamic and static point clouds into a separate set P = {p 1 , p 2 , ..., pn}, all points are considered as potential classes The DBSCAN algorithm aims to minimize the number of clusters while ensuring that each pair of points within the cluster is density-reachable, which conforms to the original properties of clustering, namely "maximum" and "connectivity". For example, the DBSCAN algorithm is expressed as:

[0072]

[0073] Among them, C represents the set of all clusters, C i represents the i-th cluster, ∈ represents the neighborhood radius, d db (p, q) represents the minimum density connectable ∈ between two points p and q. Different objects or targets can be clustered by the DBSCAN algorithm to handle noise and artifacts.

[0074] Step S132: for the clustering results, anchor point denoising based on micro-Doppler

[0075] After obtaining the clustering results, the center c of the static point cloud obtained based on micro-motion Doppler is used as the anchor point, and an axis-aligned bounding box B(c, l, w, h) is set, where c, l, w and h represent the center, length, width and height of the AABB (axis-aligned bounding box), respectively. Since the point cloud obtained based on micro-motion Doppler is based on human micro-motion, it can be effectively distinguished from static objects. Finally, the dynamic and static point clouds from the human target can be obtained.

[0076] Step S140, based on the fused clustered point cloud data, continuous positioning and tracking of multiple targets is achieved.

[0077] In this step, based on the fused and clustered point cloud data, continuous positioning and tracking of multiple targets are achieved, including accurate extraction and tracking of target position, motion trajectory and other information.

[0078] In one embodiment, continuous positioning and tracking of multiple targets includes the following steps:

[0079] Step S141, trajectory prediction.

[0080] In the trajectory prediction step, the center of mass of the trajectory at time n is predicted based on the motion tracking model. The predicted center of mass is marked as G apr (n).

[0081] Specifically, for each trackable target, the Kalman filter is used to estimate the tracking group centroid at time n based on the state and process covariance matrix estimated at time n-1. For each trackable object, the prior state S is calculated. apr (n) and covariance estimate P apr (n), expressed as:

[0082] S apr (n) = Fs(n-1) (4)

[0083] P apr (n) = FP(n-1)F T +Q(n-1) (5)

[0084] Among them, Q(n) is the process noise covariance matrix, F represents the state transfer matrix for predicting the next state value, s represents the state quantity, P represents the covariance matrix, and Q represents the noise brought by the prediction model itself.

[0085] Step S142: track association

[0086] The track association step involves associating the radar measurement with an existing track. In this step, a threshold is calculated for each track target. Then the Mahalanobis distance between each measurement point and each track is calculated. If the distance between the point and a track is less than the established threshold, the weight value of the track for the point is calculated. Finally, each point will be associated with the track with the highest corresponding weight value in descending order.

[0087] Step S143, track allocation

[0088] For measurement points that are not associated with any track implementation, a new track object is allocated and initialized.

[0089] First, a measurement point is randomly selected as the main measurement point, and the center of mass is set equal to its (range / angle). For the remaining measurement points, first check whether they fall within the measurement criteria (speed check, distance check). If they meet the requirements, recalculate the center of mass and add it to the cluster. After completing the detection of all measurement points, determine whether the new group contains enough measurement points, signal-to-noise ratio and minimum dynamic amount. If the criteria are met, create a new tracking object for it.

[0090] Step S144, track update

[0091] In one embodiment, the tracker update step includes:

[0092] 1) For each track, first calculate the average of all associated measurements

[0093] 2) Calculate the update vector based on the average value of all associated measurements, expressed as:

[0094]

[0095] Among them, H(S apr (n)) represents the observation matrix, and y(n) represents the vector of update quantities.

[0096] 3) Calculate the update residual covariance matrix, expressed as:

[0097]

[0098] Among them, J H represents the transformation matrix, R c is the Kalman updated centroid measurement error covariance matrix, expressed as:

[0099]

[0100] in, is the trajectory discrete matrix, R m is the measurement covariance matrix, N A is the number of measurements associated with a given trajectory, is the estimated number of elements in the target tracked by a given trajectory, Is The weight factor given indicates the case where there is no false detection, that is,

[0101] From the above formula, we can see that R c is the sum of two factors. The first term represents the error in the centroid measurement due to radar measurement errors. This error decreases as the number of measurements associated with a given track increases. The second term represents the uncertainty (since not all elements can be observed). The estimated number of elements in the track The recursive estimate is:

[0102]

[0103] Among them, α N Representation weight

[0104] 4) Trajectory discrete matrix estimation D at time n n As described below. For the measurement vector There are discrete matrices:

[0105]

[0106] Among them, each element in D represents the statistical weighted distance, r(n) represents the distance, represents the azimuth angle, θ(n) represents the elevation angle, Indicates speed, and and is the average value calculated for a given set of measurements. Based on this, the discrete matrix can be updated as:

[0107]

[0108] Among them, α D represents weight, and D represents the statistical weighted distance matrix.

[0109] 5) Calculate the Kalman gain K(n):

[0110]

[0111] Among them, C(n) represents the update residual covariance matrix.

[0112] 6) Calculate the posterior state vector s(n)

[0113] s(n)=S apr (n)+K(n)y(n) (13)

[0114] 7) Calculate the posterior error covariance matrix P(n):

[0115] P(n)=P apr (n)-K(n)J H (S apr (n))P apr (n)) (14)

[0116] Correspondingly, the present invention also provides a millimeter-wave-based continuous positioning and tracking system for dynamic and static targets, which is used to implement one or more aspects of the above method. For example, the system includes: a point cloud data acquisition module: used to use a millimeter-wave radar sensor to obtain dynamic point cloud data and static point cloud data reflected by the target, the dynamic point cloud data reflects the target motion state information, and the static point cloud data reflects the target's vital signs information in a static state; a point cloud fusion clustering module: used to fuse the dynamic point cloud data and the static point cloud data to obtain fused point cloud data, and cluster the fused point cloud data with the goal of distinguishing the reflection points between the target and the environment to obtain clustering results; a target tracking module: used to perform continuous positioning and tracking of multiple targets based on the clustering results to obtain tracking results. Each module in the system can be implemented using a general-purpose processor, a dedicated processor or an FPGA.

[0117] In summary, compared with the prior art, the present invention has the following advantages:

[0118] 1) Traditional methods have limited effects when processing static targets and cannot effectively distinguish the difference between static targets and the surrounding environment, resulting in a decrease in the accuracy of continuous positioning. The present invention can extract rich point cloud information from static targets to achieve continuous tracking and positioning of targets, thereby improving the processing efficiency and accuracy of static targets.

[0119] 2) Traditional methods have difficulty distinguishing the human body from the surrounding environment and are easily disturbed by static objects. The present invention uses the center of the point cloud based on micro-Doppler as an anchor point to better distinguish the human body from the surrounding environment and improve the accuracy of target recognition and positioning.

[0120] 3) Traditional methods may be interfered by noise and static clutter during target tracking, resulting in reduced tracking accuracy. However, the present invention proposes to combine dynamic and static point cloud information to enrich human body point cloud information, which can more accurately track the position and motion state of the target, and improve the accuracy and stability of target tracking.

[0121] 4) The present invention can effectively extract the dynamic and static information of the target, realize the continuous positioning and tracking of multiple targets, and has high practicality and application value, such as being applied to indoor human positioning and tracking and human-computer interaction scenarios. In the commercial field, such as shopping malls, museums and other public places, it can provide convenient navigation and guide services to enhance user experience; in the medical field, it can realize the precise location of patients, medical staff, and medical equipment, and improve the efficiency and quality of medical services; in the public safety field, it can assist rescue operations; in the smart home field, it can realize intelligent control and personalized services of home appliances.

[0122] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0123] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0124] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0125] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0126] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0129] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0130] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for continuous positioning and tracking of dynamic and static targets based on millimeter waves, comprising the following steps: Using a millimeter wave radar sensor to obtain dynamic point cloud data and static point cloud data reflected by the target, the dynamic point cloud data reflects the target's motion state information, and the static point cloud data reflects the target's vital sign information in a static state; The dynamic point cloud data and the static point cloud data are fused to obtain fused point cloud data, and the fused point cloud data are clustered using a density-based clustering algorithm with the goal of distinguishing reflection points between a target and an environment to obtain a clustering result; Based on the clustering results, continuous positioning and tracking of multiple targets are performed to obtain tracking results.

2. The method according to claim 1, characterized in that The dynamic point cloud data is obtained according to the following steps: Millimeter wave original signal acquisition, for the target with a distance d from the radar, collect the original millimeter wave signal and get N s ×N c ×N a The intermediate frequency complex real signal matrix M, where N s Indicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame, N a Indicates the number of transmit-receive pairs; Perform distance-dimensional Fourier transform on the signal matrix M and obtain a size of Matrix M R ,in is the length of the distance-dimensional Fourier transform; For the matrix M R Averaging in the second dimension gives the static clutter matrix The matrix M R and the static clutter matrix After subtraction, the matrix for removing static clutter is obtained Pair Matrix Applying the Doppler Fourier transform in the second dimension gives The distance-Doppler-antenna matrix M RDA ,in, is the length of the Doppler Fourier transform; For the range-Doppler-antenna matrix M RDA After summing in the third dimension, we get the range-Doppler heat map M RD , traverse the pixel points in the range-Doppler thermal map, and if the energy value of the pixel is higher than the set first energy threshold, mark it as a high energy point p i =(r, v), thus obtaining the set P of all high energy points d ={p1, p2, ..., p n }, where r represents the distance of the point, v represents the Doppler velocity, and n represents the total number of high-energy points; Perform the Fourier transform of the azimuth angle and the Fourier transform of the elevation angle on each current high energy point to obtain the azimuth angle Δφ a and the pitch angle Δφ e , the high energy point set P d The information of the point in is updated to pi = (r, Δφ a , Δφ e ,v),Dynamic point cloud data as target reflection.

3. The method according to claim 1, characterized in that The static point cloud data is obtained according to the following steps: For a target with a distance d from the radar, collect the original millimeter wave signal and get N s ×N c ×N a The intermediate frequency complex real signal matrix M, where N s Indicates the number of sampling points for each chirp wave, N c Indicates the number of chirp cycles in a frame, N a Indicates the number of transmit-receive pairs; Perform distance-dimensional Fourier transform on the signal matrix M and obtain a size of Matrix M R ,in is the length of the distance-dimensional Fourier transform; For the matrix M R Averaging in the second dimension gives the static clutter matrix For the static clutter matrix By continuously collecting multiple frames of static clutter and superimposing these data frames one by one, a The matrix M sc , where N f is the frame number of the data frame; For the matrix M sc Averaging in the first dimension yields the static clutter matrix The matrix M sc and the static clutter matrix After subtraction, the matrix after removing static clutter is obtained Pair Matrix After applying the micro-Doppler FFT on the first dimension, we obtain The matrix M RMMD ,in is the length of the micro-Doppler FFT; For the matrix M RMMD After summing in the third dimension, we get Distance-Micro Motion Doppler Heat Map Traversing the pixel points of the distance-micro-motion Doppler thermal map, if the energy value of the pixel is higher than the set second energy threshold, marking it as a high energy point, thereby obtaining a set of all high energy points; Perform the Fourier transform of the azimuth angle and the Fourier transform of the elevation angle on each current high energy point to obtain the azimuth angle Δφ a and the pitch angle Δφ e , update the information of the points in the high energy point set to p i =(r, Δφ a , Δφ e ,v),Dynamic point cloud data as target reflection.

4. The method according to claim 1, characterized in that: Clustering the fused point cloud data is performed according to the following steps: Merge the dynamic and static point clouds into a single set P = {p1, p2, ..., p n }, all points are considered as potential classes And minimize the number of clusters to obtain the clustering result, expressed as: Among them, C represents the set of all clusters, C i represents the i-th cluster, ∈ represents the neighborhood radius, d db (p, q) represents the minimum density-connectable ∈ between two points p and q; For the clustering results, the center c of the static point cloud obtained based on micro-Doppler is used as the anchor point for denoising, and an axis-aligned bounding box B(c, l, w, h) is set, where c, l, w and h represent the center, length, width and height of the axis-aligned bounding box, respectively, to obtain dynamic and static point clouds from human targets.

5. The method according to claim 4, characterized in that Multi-target continuous positioning and tracking includes the following steps: Trajectory prediction step: For each tracked target, use the Kalman filter to estimate the tracking group centroid at time n based on the state and process covariance matrix estimated at time n-1, where for each tracked object, the prior state S is calculated apr (n) and covariance estimate P apr (n), expressed as: S apr (n)=Fs(n-1) P apr (n)=FP(n-1)F T +Q(n-1) Among them, Q(n) is the process noise covariance matrix, F represents the state transfer matrix for predicting the next state value, s represents the state quantity, P represents the covariance matrix, and Q represents the noise caused by the prediction; Trajectory association step: Calculate the Mahalanobis distance between each measurement point and each trajectory. If the distance between the point and a trajectory is less than the set threshold, calculate the weight value of the trajectory for the point, and associate each point with the trajectory with the highest corresponding weight value in descending order. Trajectory assignment step: randomly select a measurement point as the main measurement point, set the centroid equal to it, and for the remaining measurement points, first check whether they fall within the measurement criteria. If they do, recalculate the centroid and add it to the cluster; Trajectory update step: For the assigned tracked objects, their trajectory measurements are periodically updated according to the following steps.

6. The method according to claim 5, characterized in that The trajectory updating step comprises: For each track, the mean of all association measures is calculated The update vector is calculated based on the average of all associated measurements and is expressed as: Calculate the update residual covariance matrix, expressed as: Among them, J H represents the transformation matrix, R c is the Kalman updated centroid measurement error covariance matrix, expressed as: Update the estimated trajectory discrete matrix at time n according to the following formula: Calculate the Kalman gain K(n): Calculate the posterior state vector s(n): s(n)=S spr (n)+K(n)y(n) Calculate the posterior error covariance matrix P(n): P(n)=P apr (n)-K(n)J H (S apr (n))P apr (n)) Among them, H(S apr (n)) represents the observation matrix, y(n) represents the vector of update quantities, is the estimated trajectory discrete matrix, R m is the measurement covariance matrix, N A is the number of measurements associated with a given trajectory, is the estimated number of elements in the target tracked by a given trajectory, Is Given the weight factor, α D represents weight, D represents statistical weighted distance matrix, C (n) Represents the update residual covariance matrix.

7. The method according to claim 6, characterized in that The estimated number of elements in the target tracked by the trajectory is estimated recursively according to the following formula: Among them, α N Represents weight.

8. A millimeter wave based dynamic and static target continuous positioning and tracking system, comprising: Point cloud data acquisition module: used to use the millimeter wave radar sensor to acquire dynamic point cloud data and static point cloud data reflected by the target, wherein the dynamic point cloud data reflects the target motion state information, and the static point cloud data reflects the target's vital signs information in a static state; Point cloud fusion clustering module: used to fuse the dynamic point cloud data with the static point cloud data to obtain fused point cloud data, and to cluster the fused point cloud data using a density-based clustering algorithm with the reflection points that distinguish the target from the environment to obtain a clustering result; Target tracking module: used to perform continuous positioning and tracking of multiple targets based on the clustering results to obtain tracking results.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images