Multi-target trajectory tracking method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-target trajectory tracking method based on millimeter wave radars. The multi-target trajectory tracking method comprises the following steps: acquiring a plurality of point cloud data from a plurality of radars; determining global coordinates of the plurality of pieces of point cloud data according to the position when the radar collects the point cloud data; performing fusion processing on the global coordinates of the point cloud data based on the starting timestamps of the plurality of radar acquisition point cloud data to obtain a point cloud data set; mapping global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain multiple pieces of two-dimensional data; classifying the plurality of two-dimensional data by using a clustering algorithm to obtain a plurality of clusters; and filtering the plurality of clusters to obtain a plurality of target positions.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology and can be applied to the fields of medical devices and indoor environment perception. In particular, it relates to a multi-target trajectory tracking method, device, electronic device, computer-readable storage medium, and computer program product based on millimeter-wave radar. Background Art

[0002] Target trajectory tracking is an important technology for ambient intelligence perception and daily activity recognition. In indoor scenarios, technologies such as cameras, Wireless Fidelity (WIFI), Radio Frequency Identification (RFID), and Inertial Measurement Unit (IMU) are used for target tracking. Cameras have high accurate tracking accuracy, but there are privacy leakage problems and are vulnerable to environmental light in indoor scenarios. WIFI is an indoor device that can be seen everywhere, but the detection accuracy and stability of WIFI are affected by factors such as signal strength and bandwidth limitations. Technologies based on RFID and IMU require uniquely identified devices to be worn on the human body, and long-term wearing brings a bad user experience. Summary of the Invention

[0003] In view of this, the present disclosure provides a multi-target trajectory tracking method, device, equipment, medium, and program product based on millimeter-wave radar.

[0004] One aspect of the present disclosure provides a multi-target trajectory tracking method based on millimeter-wave radar, including: obtaining multiple point cloud data from multiple radars; determining the global coordinates of the multiple point cloud data according to the positions of the radars when collecting the point cloud data; performing fusion processing on the global coordinates of the point cloud data based on the start timestamps of the multiple radars collecting the point cloud data to obtain a set of point cloud data; mapping the global coordinates of the point cloud data in the set of point cloud data to a two-dimensional plane to obtain multiple two-dimensional data; using a clustering algorithm to classify the multiple two-dimensional data to obtain multiple clusters; and performing filtering processing on the multiple clusters to obtain multiple target positions.

[0005] According to an embodiment of the present disclosure, determining the global coordinates of the multiple point cloud data according to the positions of the radars when collecting the point cloud data includes: mapping the point cloud data from the polar coordinate system to the rectangular coordinate system based on the height of the radar when collecting the point cloud data and the angle relative to a preset position; determining the translation transformation matrix of the radar based on the position of the radar; and mapping the point cloud data from the rectangular coordinate system to the global coordinate system based on the translation transformation matrix to obtain the global coordinates of the multiple point cloud data.

[0006] According to an embodiment of the present disclosure, the point cloud data is multi-frame point cloud data, and each frame of point cloud data has a timestamp; the global coordinates of the point cloud data are fused based on the start timestamps of the point cloud data collected by multiple radars, and the obtained point cloud data set includes: determining a target start timestamp from the start timestamps of the point cloud data obtained from multiple radars, where the start timestamp represents the timestamp corresponding to the first frame of point cloud data collected by the radar; for the multi-frame point cloud data collected by each radar, using the target start timestamp as the new start timestamp, and fusing the global coordinates of the point cloud data frame by frame.

[0007] According to an embodiment of the present disclosure, the point cloud data includes velocity data; the method further includes, before mapping the global coordinates of the point cloud data in multiple point cloud data sets to a two-dimensional plane to obtain multiple two-dimensional data: removing the global coordinates of the point cloud data from the point cloud data set when the velocity data corresponding to the global coordinates of the point cloud data is zero.

[0008] According to an embodiment of the present disclosure, filtering multiple clusters to obtain multiple target positions includes: filtering multiple clusters based on the unscented Kalman filter to obtain multiple target positions.

[0009] According to an embodiment of the present disclosure, the method further includes: obtaining a distance matrix based on the distance between each target position and the historical trajectory of each target; determining the target position corresponding to each historical trajectory based on the minimum distance cost of the distance matrix; and associating the historical trajectory and the target position corresponding to the historical trajectory.

[0010] Another aspect of the present disclosure provides a multi-target trajectory tracking device based on a millimeter-wave radar, including: an acquisition module, a determination module, a fusion module, a mapping module, a classification module, and a filtering module. The acquisition module is used to acquire multiple point cloud data from multiple radars. The determination module is used to determine the global coordinates of the multiple point cloud data according to the positions of the radars when collecting the point cloud data. The fusion module is used to fuse the global coordinates of the point cloud data based on the start timestamps of the point cloud data collected by multiple radars to obtain a point cloud data set. The mapping module is used to map the global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain multiple two-dimensional data. The classification module is used to classify the multiple two-dimensional data using a clustering algorithm to obtain multiple clusters. The filtering module is used to filter the multiple clusters to obtain multiple target positions.

[0011] Another aspect of the present disclosure provides an electronic device, including: one or more processors; a storage device for storing one or more programs, where when the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the above-mentioned multi-target trajectory tracking method based on a millimeter-wave radar.

[0012] Another aspect of the present disclosure also provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor is caused to execute the above-mentioned multi-target trajectory tracking method based on a millimeter-wave radar.

[0013] Another aspect of the present disclosure also provides a computer program product, including a computer program, which when executed by a processor implements the above-mentioned multi-target trajectory tracking method based on a millimeter-wave radar.

[0014] According to an embodiment of the present disclosure, multi-target trajectory tracking based on a millimeter-wave radar does not need to worry about privacy issues, is not affected by lighting conditions, has the characteristics of non-contact monitoring, and can ensure the accuracy of target trajectory tracking.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Through the following description of the embodiments of the present disclosure with reference to the drawings, the above and other objects, features and advantages of the present disclosure will become clearer. In the drawings:

[0017] Figure 1 Schematically shows an application scenario diagram of a multi-target trajectory tracking method, device, electronic device, computer-readable storage medium and computer program product based on a millimeter-wave radar according to an embodiment of the present disclosure;

[0018] Figure 2 Schematically shows a flowchart of a multi-target trajectory tracking method based on a millimeter-wave radar according to an embodiment of the present disclosure;

[0019] Figure 3 Schematically shows a flowchart of a multi-target trajectory tracking method based on a millimeter-wave radar according to another embodiment of the present disclosure;

[0020] Figure 4 Schematically shows a structural block diagram of a multi-target trajectory tracking device based on a millimeter-wave radar according to an embodiment of the present disclosure;

[0021] Figure 5 Schematically shows a structural block diagram of a multi-target trajectory tracking device based on a millimeter-wave radar according to another embodiment of the present disclosure; and

[0022] Figure 6 Schematically shows a schematic block diagram of an electronic device suitable for implementing a multi-target trajectory tracking method based on a millimeter-wave radar according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Various details of the embodiments of the present disclosure are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.

[0024] In the technical solution of the present disclosure, the processing of data involved (such as including but not limited to user personal information) in terms of collection, storage, use, processing, transmission, provision, disclosure, and application, etc., all comply with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and it does not violate public order and good customs.

[0025] The terms used herein are merely for describing specific embodiments and are not intended to limit the present disclosure. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those of ordinary skill in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0028] Embodiments of the present disclosure provide a multi-target trajectory tracking method. The method includes: acquiring a plurality of point cloud data from a plurality of radars; determining the global coordinates of the plurality of point cloud data according to the positions of the radars when collecting the point cloud data; performing fusion processing on the global coordinates of the point cloud data based on the start timestamps of the point cloud data collected by the plurality of radars to obtain a point cloud data set; mapping the global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain a plurality of two-dimensional data; classifying the plurality of two-dimensional data using a clustering algorithm to obtain a plurality of clusters; and performing filtering processing on the plurality of clusters to obtain a plurality of target positions.

[0029] Adopting the above solution, the multi-target trajectory tracking based on millimeter-wave radar does not need to worry about privacy issues, is not affected by lighting conditions, has the characteristics of non-contact monitoring, and can ensure the accuracy of target trajectory tracking.

[0030] Figure 1 FIG. schematically shows an application scenario diagram of a multi-target trajectory tracking method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a millimeter-wave radar according to an embodiment of the present disclosure.

[0031] As Figure 1 shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0033] The server 105 may be a server providing various services, such as a background management server (only an example) that supports the websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process data such as received user requests, etc., and feedback the processing results (such as web pages, information, or data obtained or generated according to user requests, etc.) to the terminal devices.

[0034] It should be noted that the multi-target trajectory tracking method provided by the embodiments of the present disclosure can generally be executed by the server 105. Correspondingly, the multi-target trajectory tracking apparatus provided by the embodiments of the present disclosure can generally be set in the server 105. The multi-target trajectory tracking method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Correspondingly, the multi-target trajectory tracking apparatus provided by the embodiments of the present disclosure can also be set in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0035] It should be understood, Figure 1The numbers of terminal devices, networks, and servers in it are merely illustrative. According to implementation requirements, there can be any number of terminal devices, networks, and servers.

[0036] Based on the Figure 1 scenario described below, the image recognition method of the disclosed embodiments will be described in detail through Figures 2 to 3 the following.

[0037] Figure 2 FIG. schematically shows a flowchart of a multi-target trajectory tracking method based on a millimeter-wave radar according to an embodiment of the present disclosure.

[0038] As Figure 2 shown, this embodiment includes operations S210 to S260.

[0039] In operation S210, multiple point cloud data from multiple radars are acquired.

[0040] For example, the radar can be a frequency-modulated millimeter-wave radar, and the point cloud data can include elevation data, azimuth data, velocity (doppler_v) data, and distance (r) data of a human target relative to the radar.

[0041] Specifically, the radar can periodically transmit a linearly frequency-modulated pulse signal and measure the reflected signal at the receiving antenna of the radar. The received signal and the transmitted signal can enter a mixer to obtain an intermediate-frequency signal. The radial distance of the target can be obtained by performing a fast Fourier transform (FFT) transformation on the intermediate-frequency signal in the distance dimension. By calculating the FFT transformation of the intermediate-frequency signal in the antenna dimension and passing through a constant false-alarm rate (CFAR) and Capon estimation, the azimuth angle and elevation angle of the target relative to the radar can be obtained. By calculating the FFT transformation in the linearly frequency-modulated pulse dimension, the mapping relationship between the phase and velocity of the continuous pulse signal can be obtained, thereby realizing the calculation of the velocity.

[0042] Based on the above operations, the point cloud data in the polar coordinate system of each radar can be obtained.

[0043] In operation S220, according to the positions of the radars when collecting the point cloud data, the global coordinates of the multiple point cloud data are determined.

[0044] The point cloud data can include elevation angle data, azimuth angle data, velocity data, and distance data. The positions of the radars when collecting the point cloud data can include the height of the radars relative to the indoor ground when collecting the point cloud data and the angle relative to a preset position.

[0045] Specifically, based on the pitch angle data, azimuth angle data, distance data in the point cloud data, as well as the height and the angle relative to the preset position when the radar acquires the point cloud data, the point cloud data can be mapped from the polar coordinate system to the rectangular coordinate system.

[0046] In order to fuse the point cloud data acquired by multiple radars, a translation transformation matrix can be determined based on the positions of the radars. It can be understood that each radar corresponds to a translation transformation matrix. For the point cloud data acquired by each radar, based on the translation transformation matrix corresponding to the radar, the point cloud data is mapped from the local rectangular coordinate system of the radar to the global coordinate system, and the global coordinates of multiple point cloud data are obtained.

[0047] In operation S230, based on the start timestamps of the point cloud data acquired by multiple radars, the global coordinates of the point cloud data are fused to obtain a set of point cloud data.

[0048] The start timestamp of the radar acquiring the point cloud data can be the timestamp when the radar acquires the first frame of point cloud data. Specifically, the timestamps of the point cloud data acquired by multiple radars can be stored in multiple lists, that is, the timestamps of multiple frames of point cloud data acquired by one radar correspond to a timestamp list, and the timestamps in each timestamp list can be arranged in chronological order. Therefore, the target start timestamp can be determined from the start timestamps of the point cloud data acquired by multiple radars, that is, the later start timestamp among multiple start timestamps.

[0049] After determining the target start timestamp, for all radars, with the target start timestamp as the new start timestamp, the global coordinates of the point cloud data are fused frame by frame.

[0050] In operation S240, the global coordinates of the point cloud data in the set of point cloud data are mapped to a two-dimensional plane to obtain multiple two-dimensional data.

[0051] The global coordinates of the point cloud data can be three-dimensional global coordinates, specifically the three-dimensional global coordinates in the x - y - z plane. Mapping the global coordinates of the point cloud data to a two-dimensional plane can be mapping to the x - y plane, thereby obtaining two-dimensional data in the x - y plane.

[0052] In operation S250, a clustering algorithm is used to classify the multiple two-dimensional data to obtain multiple clusters.

[0053] For example, the clustering algorithm can be a density clustering algorithm. Specifically, the neighborhood radius and the minimum number of points for clustering can be preset in advance. When using the density clustering algorithm to classify multiple two-dimensional data into multiple clusters, it can start from any unvisited two-dimensional data, calculate the amount of data within the neighborhood of the two-dimensional data. If the amount of data within the neighborhood of the two-dimensional data is greater than or equal to the preset minimum number of points, the points within this area can be regarded as a cluster, and then continue to search for new high-density areas within the neighborhoods of these points until no new high-density areas can be found. Then continue to search for the next unvisited point and repeat the above process until all points have been visited.

[0054] Specifically, when clustering multiple two-dimensional data, it can be based on the global coordinates of the point cloud data corresponding to the two-dimensional data, and segmented clustering processing is adopted in the z-axis direction. The distance evaluation method between two two-dimensional data ( , ) and ( , ) is as shown in formula (1).

[0055] Formula (1)

[0056] Where, ( , ) and ( , ) respectively represent the coordinates of two two-dimensional data, represents the z-axis positions of the global coordinates corresponding to these two two-dimensional data respectively and , represents a region in the z-axis direction.

[0057] In operation S260, filter processing is performed on multiple clusters to obtain multiple target positions.

[0058] For example, each cluster corresponds to a target to be detected. The unscented Kalman filter can be used to perform filter processing on multiple clusters to obtain multiple target positions.

[0059] The multi-target trajectory tracking based on millimeter-wave radar proposed in the embodiments of the present disclosure does not need to worry about privacy issues, is not affected by lighting conditions, has the characteristics of non-contact monitoring, and performing filter processing on clusters can avoid the uncertainty of radar measurement of targets and ensure the accuracy of target trajectory tracking.

[0060] According to another embodiment of the present disclosure, determining the global coordinates of multiple point cloud data based on the position when the radar acquires the point cloud data includes: mapping the point cloud data from the polar coordinate system to the rectangular coordinate system based on the height when the radar acquires the point cloud data and the angle relative to the preset position; determining the translation transformation matrix of the radar based on the position of the radar; mapping the point cloud data from the rectangular coordinate system to the global coordinate system based on the translation transformation matrix to obtain the global coordinates of the multiple point cloud data.

[0061] For multiple radar nodes deployed indoors, the point cloud data acquired by each radar is in the local polar coordinate system relative to the radar itself. Therefore, first, the point cloud data in the local polar coordinate system relative to the radar itself is converted into the point cloud data in the local rectangular coordinate system, and then the point cloud data is mapped from the local rectangular coordinate system of the radar to the global coordinate system. The transformation formula for converting the point cloud data in the polar coordinate system to the rectangular coordinate system is shown in Formula (2):

[0062] Formula (2)

[0063] Among them, the point cloud data in the polar coordinate system of each radar itself can be (elevation, azimuth, doppler_v, r), where elevation represents the pitch angle data, azimuth represents the azimuth angle data, doppler_v represents the velocity data, and r represents the distance data. represents the angle relative to the preset position when the radar acquires the point cloud data. represents the height when the radar acquires the point cloud data.

[0064] In order to enable the point cloud data acquired by multiple radars to be fused and processed, in the case of n radars, the position of a single point cloud of the kth radar in the radar coordinate system can be expressed as Pp = [xk, yk, zk], and the position translation transformation matrix of the radar is M = [fset_xk, fset_yk, 0]. Then, the conversion from the radar coordinate system to the global coordinate system is expressed as Gp = Pp + M. The conversion of the local coordinates of the point cloud to the global coordinate system is shown in Formula (3):

[0065] Formula (3)

[0066] Among them, xk, yk, and zk are the x-axis, y-axis, and z-axis coordinates of the point cloud data in the local rectangular coordinate system of the radar, respectively, and [fset_xk, fset_yk, 0] is the translation transformation matrix of the radar.

[0067] It can be understood that by mapping the point cloud data collected by multiple radars from the local polar coordinate system to the local rectangular coordinate system and then from the local rectangular coordinate system to the global coordinate system, the point cloud data collected by different radars can be fused, the differences caused by different coordinate systems can be eliminated, more complete information can be obtained, and the accuracy of target detection and tracking can be improved.

[0068] According to another embodiment of the present disclosure, the point cloud data is multi-frame point cloud data, and each frame of point cloud data has a timestamp; based on the start timestamps of the point cloud data collected by multiple radars, the global coordinates of the point cloud data are fused to obtain a set of point cloud data, including: determining a target start timestamp from the start timestamps of obtaining the point cloud data from multiple radars, where the start timestamp represents the timestamp corresponding to the first frame of point cloud data collected by the radar; for the multi-frame point cloud data collected by each radar, using the target start timestamp as the new start timestamp, and fusing the global coordinates of the point cloud data frame by frame.

[0069] For example, assume that the data sampling rate of each radar is 20 Hz, that is, each radar collects 20 frames of point cloud data per second. Each frame of point cloud data can record the time as the timestamp of the point cloud data collected by the radar. The specific timestamp can be accurate to milliseconds. The timestamps of the multi-frame point cloud data can be stored in a list form, that is, the timestamps of the multi-frame point cloud data collected by one radar correspond to a timestamp list, and the timestamps in each timestamp list can be arranged in chronological order.

[0070] The start timestamp of the point cloud data collected by the radar can be the timestamp of the first frame of point cloud data collected by the radar, and the target start timestamp can be the later start timestamp among the multiple start timestamps. Specifically, according to the chronological order of the start timestamps in each timestamp list, determine the later start timestamp, that is, the target timestamp.

[0071] For multiple radars, based on the target start timestamp, respectively search for the timestamp index closest to this benchmark among the timestamps of the point cloud data collected by each radar, use the timestamp at the index position as the new start timestamp, and fuse the global coordinates of the point cloud data according to formula (4).

[0072] Formula (4)

[0073] Where t represents the timestamp, Gp0_t1 represents the global coordinate of the point cloud data collected by the 0th radar at time t1, and so on, Gpk_tT represents the global coordinate of the point cloud data collected by the kth radar at time T.

[0074] It can be understood that fusing the global coordinates of the point cloud data based on the start timestamps of the point cloud data collected by multiple radars can ensure that the point cloud data collected by multiple radars can be strictly used for fusion processing.

[0075] According to another embodiment of the present disclosure, the point cloud data includes velocity data; the multi-target trajectory tracking method further includes, before mapping the global coordinates of the point cloud data in a plurality of point cloud data sets to a two-dimensional plane to obtain a plurality of two-dimensional data: when the velocity data corresponding to the global coordinates of the point cloud data is zero, removing the global coordinates of the point cloud data from the point cloud data set.

[0076] Since the point cloud data collected by the radar based on the reflection signals of static objects will interfere with the tracking of the target trajectory, it is necessary to remove this part of the point cloud data. This part of the point cloud data that will interfere with the tracking of the target trajectory can be called noise. To remove the point cloud data collected by the radar based on the reflection signals of static objects, such as furniture and walls in a room, it can be assumed that these static objects will not move continuously in a short period of time, and the point cloud data collected by the radar based on the reflection signals of static objects and dynamic objects can be distinguished by the velocity characteristics of the point cloud data.

[0077] For example, when the velocity data corresponding to the global coordinates of the point cloud data is zero, the global coordinates of the point cloud data can be removed from the point cloud data set, that is, the noise is removed.

[0078] For the point cloud data set obtained after removing the noise, voxel processing of the point cloud data can also be adopted. Voxel processing is to divide the global coordinates of the point cloud data by the voxel radius. The voxel radius can be set to a predefined threshold TH1. It can be understood that voxel processing can remove clutter point cloud data. Clutter point cloud data generally appears disordered and not aggregated, and at the same time reduces the computational complexity.

[0079] In order to better reflect the target characteristics at a certain moment, for the voxelized point cloud data set, a processing step of accumulating consecutive frame point cloud data can be adopted, that is, by splicing consecutive frame point cloud data to form multi-frame point cloud data to jointly represent the target at a certain moment.

[0080] It can be understood that removing the noise can ensure the accuracy of the point cloud data, reduce the interference of static objects on the target trajectory tracking, and provide a high-quality data basis for subsequent target position positioning and target trajectory tracking.

[0081] According to another embodiment of the present disclosure, filtering a plurality of clusters to obtain a plurality of target positions includes: filtering a plurality of clusters based on the unscented Kalman filter to obtain a plurality of target positions.

[0082] In an actual scenario, since the position and speed of a human target change, the existing technology generally constructs an acceleration model of human motion and uses this acceleration model to predict the target position. The specific acceleration model is shown in formula (5):

[0083] Formula (5)

[0084] Among them, the state variables of the human target are expressed as , where x1 and x2 respectively represent the positions of the human target on the x-axis and y-axis. Correspondingly, represents the speed of the target, , represents the acceleration of the target, and represent the new position predicted by the model.

[0085] However, in an actual application scenario, the trajectory of the human target's motion is non-linear. Therefore, this method of solely using the acceleration model to predict the target position is not accurate.

[0086] In this application, tracking filtering processing is performed through the unscented Kalman filtering processing method to obtain the target prediction position. First, sampling points ( s) are selected to best represent the mean and covariance of the true state distribution. The specific point sampling method can be the Van der Merwe's method. NK points can be selected from multiple data points to represent the true state distribution. Specifically, the distribution range of the sampling points around the mean can be set to TH2. The weight parameter involved in the Van der Merwe's method can be represented by beta, and can be specifically set to TH3. The secondary scaling parameter can be represented by Kappa, and can be specifically set to TH4.

[0087] These sampling points are passed through the non-linear state transition function f, as shown in formula (6):

[0088] Formula (6)

[0089] Among them, s represents the selected sampling points, f represents the non-linear state transition function, represents the value obtained by passing the sampling points through the non-linear state transition function.

[0090] In the prediction stage, the mean of the distribution and the covariance are calculated respectively. The calculation method of the mean is shown in formula (7):

[0091] Formula (7)

[0092] Among them, represents the weight of the mean, specifically a preset value, represents the value obtained by the sampling point through the non - linear state transition function, and NK represents the number of sampling points, is the mean, and can also be called the prior prediction mean.

[0093] Covariance is calculated as shown in formula (8):

[0094] Formula (8)

[0095] Among them, Q represents the process noise covariance matrix, which can be specifically set as a 6 * 6 random matrix with variance TH5, represents the weight of the covariance, specifically a preset value, represents the value obtained by the sampling point through the non - linear state transition function, is the mean, is the covariance, and can also be called the prior prediction covariance.

[0096] In the update stage, the measurement function h converts the sampling point into a predicted value with the same physical quantity as the measurement representation, that is, the predicted value is obtained through the measurement function h, specifically as shown in formula (9):

[0097] Formula (9)

[0098] Among them, represents the weight of the mean, specifically a preset value, h represents the measurement function, represents the value obtained by the sampling point through the non - linear state transition function, NK represents the number of sampling points, represents the predicted value.

[0099] In this application, the measurement of the radar on the target can be directly expressed as where the and are the planar projection positions of a point cloud data of a certain point of the target obtained by the radar. Therefore, the residual between the measurement value and the predicted value is calculated as shown in formula (10):

[0100] Formula (10)

[0101] Among them, z represents the true measurement of the radar on the target, represents the predicted value, represents the residual between the measurement value and the predicted value.

[0102] The new state estimate is calculated as shown in formula (11):

[0103] Equation (11)

[0104] Wherein, is the prior prediction mean, i.e., the mean calculated in Equation (7) , represents the residual between the measured value and the predicted value, and K is the calculated Kalman gain.

[0105] The new covariance is calculated as shown in Equation (12):

[0106] Equation (12)

[0107] Wherein, K is the calculated Kalman gain, is the prior prediction covariance, i.e., the covariance calculated in Equation (8) , is the measurement covariance of the measurement sampling point.

[0108] Measurement covariance is calculated as shown in Equation (13):

[0109] Equation (13)

[0110] Wherein, represents the weight of the covariance, specifically a preset value, h represents the measurement function, represents the value obtained by the sampling point through the non-linear state transition function, R represents the measurement noise function, and can be set as a 2*2 matrix with a variance of TH6.

[0111] The calculation formula of the Kalman gain K is shown in Equation (14):

[0112] Equation (14)

[0113] Wherein, NK represents the number of sampling points, represents the value obtained by the sampling point through the non-linear state transition function, represents the weight of the covariance, specifically a preset value, h represents the measurement function, represents the predicted value, represents the measurement covariance.

[0114] It can be understood that in this application, by selecting sampling points, the probability distribution of the non-linear system can be accurately approximated, and using the unscented Kalman filter to predict the target position can avoid the errors brought by the linear model, thereby more accurately tracking the true trajectory of the human target.

[0115] According to another embodiment of the present disclosure, the multi-target trajectory tracking method further includes: obtaining a distance matrix based on the distance between each target position and the historical trajectory of each target; determining the target position corresponding to each historical trajectory based on the minimum distance cost of the distance matrix; and associating the historical trajectory with the target position corresponding to the historical trajectory.

[0116] In order to associate the currently detected target position with the historical target trajectory, the Euclidean distance can be calculated based on the target position detected at the current moment and the historical trajectory to obtain a distance matrix. Specifically, each column in the matrix can represent a historical trajectory, and the number of trajectories is N. Each row in the matrix can represent the currently detected target, and the number of targets is M. Therefore, the distance matrix can be represented as a matrix of size M*N. After determining the distance matrix, the minimum distance cost of the distance matrix can be determined through the Hungarian algorithm to obtain the most suitable row detection target associated with each column of the trajectory.

[0117] In the initial stage, an Unscented Kalman Filter (UKF) tracker can be assigned to each detected target to obtain the latest prediction of each target. And the UKF of the target with successful trajectory association can be updated to obtain a new estimated position value.

[0118] For newly emerged targets, there may be a situation where they are not successfully associated with the historical trajectory. A new UKF tracker can be created for the newly detected targets. At the same time, for trajectories that have not been updated for a long time, they can be deleted if the non-update time exceeds the set number of frames.

[0119] It can be understood that the results of multi-target trajectory tracking can be connected to the backend motion rehabilitation management unit. Through the precise tracking of multiple targets, it is beneficial to scientifically manage the motion rehabilitation and motion state of personnel.

[0120] Figure 3 The flowchart of the multi-target trajectory tracking method based on millimeter-wave radar according to another embodiment of the present disclosure is schematically shown. This method includes operations S310~S380. Those skilled in the art can understand that the following embodiments are only examples, and the present disclosure is not limited thereto.

[0121] In operation S310, multiple point cloud data from multiple radars are obtained.

[0122] For example, the radar can be a millimeter-wave radar, and more specifically a frequency-modulated millimeter-wave radar. The point cloud data can include the pitch angle, azimuth angle, speed, and distance of the target relative to the radar.

[0123] In operation S320, the global coordinates of the multiple point cloud data are determined according to the positions of the radars when collecting the point cloud data.

[0124] For example, according to the pitch angle data, azimuth angle data, distance data in the point cloud data, as well as the height relative to the ground and the angle relative to the preset position when the radar collects the point cloud data, the point cloud data can be mapped from the polar coordinate system of the radar to the local rectangular coordinate system of the radar. Then, a translation transformation matrix is determined according to the position of the radar, and the point cloud data is mapped from the local rectangular coordinate system of the radar to the global coordinate system based on the translation transformation matrix, so as to obtain the global coordinates of the point cloud data.

[0125] In operation S330, based on the start timestamps of the point cloud data collected by multiple radars, the global coordinates of the point cloud data are fused to obtain a set of point cloud data.

[0126] For example, the start timestamp of the point cloud data collected by the radar can be the timestamp of the first frame of point cloud data collected by the radar. The target timestamp can be determined from the start timestamps of the point cloud data collected by multiple radars. Specifically, the target timestamp can be the maximum start timestamp. For all radars, the global coordinates of the point cloud data can be fused frame by frame with the maximum start timestamp as the new start timestamp to obtain a set of point cloud data.

[0127] In operation S340, noise is removed.

[0128] For example, when the velocity data of the point cloud data is zero, it can be considered that the point cloud data is obtained from the signal reflected by the static object by the radar, and this point cloud data is the noise, and the global coordinates of this point cloud data are removed from the set of point cloud data.

[0129] In operation S350, the global coordinates of the point cloud data in the set of point cloud data are mapped to a two-dimensional plane to obtain a plurality of two-dimensional data.

[0130] For example, the global coordinates of the point cloud data can be three-dimensional global coordinates. Mapping the global coordinates of the point cloud data to a two-dimensional plane can mean mapping the three-dimensional global coordinates of the point cloud data to the x-y plane, so as to obtain two-dimensional data in the x-y plane.

[0131] In operation S360, a clustering algorithm is used to classify the plurality of two-dimensional data to obtain a plurality of clusters.

[0132] For example, the density clustering algorithm can be used to classify the plurality of two-dimensional data to obtain a plurality of clusters.

[0133] In operation S370, the plurality of clusters are filtered to obtain a plurality of target positions.

[0134] For example, each cluster corresponds to a target to be detected, and the unscented Kalman filter can be used to filter the plurality of clusters to obtain a plurality of target positions

[0135] In operation S380, the target position and the historical trajectory are associated and managed.

[0136] For example, the Euclidean distance can be calculated based on the target position detected at the current moment and the historical trajectory to obtain a distance matrix, and then the Hungarian algorithm can be used to determine the minimum distance cost of the distance matrix to obtain the most suitable target position associated with each historical trajectory.

[0137] Figure 4 The structural block diagram of a multi-target trajectory tracking device based on a millimeter-wave radar according to an embodiment of the present disclosure is schematically shown.

[0138] As Figure 4 shown, the device 400 of this embodiment includes a radar sub-node module 410, a data forwarding module 420, a calculation and processing unit 430, and a real-time display module 440.

[0139] The radar sub-node module 410 includes a plurality of radar sub-nodes, and the plurality of radar sub-nodes can be respectively arranged at a plurality of position points in the room according to the situation. Specifically, it can be used to collect point cloud data and can also be used to transmit the point cloud data to the data forwarding module 420 through means such as network cable, WIFI, and Fourth Generation mobile communication technology (4G) module.

[0140] The data forwarding module 420 can summarize the point cloud data and transmit it to the calculation and processing unit 430 in a parallel processing manner.

[0141] The calculation and processing unit 430 can be used to run the multi-target trajectory tracking method proposed in this application and display the results in real time on the real-time display module 440 or access the backend motion rehabilitation management unit.

[0142] The real-time display module 440 can be used to display the results of target trajectory tracking in real time.

[0143] Figure 5 The structural block diagram of a multi-target trajectory tracking device based on a millimeter-wave radar according to another embodiment of the present disclosure is schematically shown.

[0144] As Figure 5 shown, the device 500 of this embodiment includes:

[0145] An acquisition module 510, configured to acquire a plurality of point cloud data from a plurality of radars. In one embodiment, the acquisition module 510 can be used to perform the operation S210 described above, which will not be elaborated here.

[0146] A determination module 520 is configured to determine the global coordinates of multiple point cloud data based on the position when the radar acquires the point cloud data. In one embodiment, the determination module 520 can be used to perform the operation S220 described above, which will not be elaborated here.

[0147] A fusion module 530 is configured to perform a fusion process on the global coordinates of the point cloud data based on the start timestamps of the multiple radar-acquired point cloud data to obtain a point cloud data set. In one embodiment, the fusion module 530 can be used to perform the operation S230 described above, which will not be elaborated here.

[0148] A mapping module 540 is configured to map the global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain multiple two-dimensional data. In one embodiment, the mapping module 540 can be used to perform the operation S240 described above, which will not be elaborated here.

[0149] A classification module 550 is configured to classify the multiple two-dimensional data using a clustering algorithm to obtain multiple clusters. In one embodiment, the classification module 550 can be used to perform the operation S250 described above, which will not be elaborated here.

[0150] A filtering module 560 is configured to perform a filtering process on the multiple clusters to obtain multiple target positions. In one embodiment, the filtering module 560 can be used to perform the operation S260 described above, which will not be elaborated here.

[0151] Figure 6 FIG. schematically shows a schematic block diagram of an electronic device suitable for implementing a multi-target trajectory tracking method based on a millimeter-wave radar according to an embodiment of the present disclosure.

[0152] As Figure 6 shown, an electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage section 608 into a random access memory (RAM) 603. The processor 601 can include, for example, a general microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 601 can also include on-board memory for caching purposes. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0153] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via the bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 602 and / or the RAM 603. It should be noted that the programs may also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 may also implement the method provided by the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0154] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, and the input / output (I / O) interface 605 is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. The drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed so that a computer program read therefrom can be installed into the storage portion 608 as needed.

[0155] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist separately without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0156] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include one or more memories other than the above-described ROM 602 and / or RAM 603 and / or ROM 602 and RAM 603.

[0157] An embodiment of the present disclosure further includes a computer program product, which includes a computer program that contains program code for executing the method shown in the flowchart. When the computer program product runs in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiment of the present disclosure.

[0158] When the computer program is executed by the processor 601, it executes the above functions defined in the system / apparatus of the embodiment of the present disclosure. According to an embodiment of the present disclosure, the above-described systems, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0159] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices and magnetic storage devices. In another embodiment, the computer program may also be transmitted and distributed in the form of a signal on a network medium, and be downloaded and installed through the communication part 609, and / or be installed from the removable medium 611. The program code contained in the computer program may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0160] In such an embodiment, the computer program may be downloaded and installed from the network through the communication part 609, and / or be installed from the removable medium 611. When the computer program is executed by the processor 601, it executes the above functions defined in the system of the embodiment of the present disclosure. According to an embodiment of the present disclosure, the above-described systems, devices, apparatuses, modules, units, etc. may be implemented by computer program modules.

[0161] It should be noted that in the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and public order and good customs are not violated. In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before obtaining or collecting user personal information.

[0162] According to the embodiments of the present disclosure, program codes for executing the computer programs provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level procedures and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, such as Java, C++, Python, the "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, by using an Internet service provider to connect through the Internet).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0164] Those skilled in the art can understand that the features described in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present disclosure.

[0165] The embodiments of the present disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art can make various substitutions and modifications, and these substitutions and modifications should fall within the scope of the present disclosure.

Claims

1. A multi-target trajectory tracking method based on millimeter wave radar, characterized in that: The method comprises: Acquire multiple point cloud data from multiple radars; Determining the global coordinates of the plurality of point cloud data according to the position of the radar when collecting the point cloud data; Based on the start timestamps of the point cloud data collected by multiple radars, the global coordinates of the point cloud data are fused to obtain a point cloud data set; Mapping the global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain a plurality of two-dimensional data; Using a clustering algorithm to classify the plurality of two-dimensional data to obtain a plurality of clusters; Filtering is performed on the multiple clusters to obtain multiple target positions.

2. The method according to claim 1, characterized in that Determining the global coordinates of the plurality of point cloud data according to the position of the radar when collecting the point cloud data comprises: Mapping the point cloud data from a polar coordinate system to a rectangular coordinate system based on the height at which the radar collects the point cloud data and the angle relative to a preset position; Determine a translation transformation matrix of the radar based on the position of the radar; The point cloud data are mapped from a rectangular coordinate system to a global coordinate system based on the translation transformation matrix to obtain global coordinates of the plurality of point cloud data.

3. The method according to claim 1, characterized in that The point cloud data is multi-frame point cloud data, each frame of point cloud data has a timestamp; the global coordinates of the point cloud data are fused based on the starting timestamps of the point cloud data collected by multiple radars to obtain a point cloud data set including: Determine a target start timestamp from start timestamps of point cloud data acquired by multiple radars, where the start timestamp represents a timestamp corresponding to a first frame of point cloud data acquired by the radar; For each frame of point cloud data collected by the radar, the target start timestamp is used as the new start timestamp, and the global coordinates of the point cloud data are fused frame by frame.

4. The method according to claim 1, characterized in that: The point cloud data includes speed data; the method further includes, before mapping the global coordinates of the point cloud data in the plurality of point cloud data sets to a two-dimensional plane to obtain a plurality of two-dimensional data: When the speed data corresponding to the global coordinates of the point cloud data is zero, the global coordinates of the point cloud data are removed from the point cloud data set.

5. The method according to claim 1, characterized in that: The filtering process for the plurality of clusters to obtain a plurality of target locations comprises: The multiple clustering filters are processed based on the unscented Kalman filter to obtain multiple target positions.

6. The method according to claim 1, characterized in that The method further comprises: A distance matrix is ​​obtained based on the distance between each target position and each target's historical trajectory; Determining a target location corresponding to each historical trajectory based on a minimum distance cost of the distance matrix; The historical track and a target location corresponding to the historical track are associated.

7. A multi-target trajectory tracking device based on millimeter wave radar, comprising: An acquisition module, used for acquiring multiple point cloud data from multiple radars; A determination module, used to determine the global coordinates of the plurality of point cloud data according to the position of the radar when collecting the point cloud data; A fusion module, configured to perform fusion processing on the global coordinates of the point cloud data based on the start timestamps of the point cloud data collected by multiple radars to obtain a point cloud data set; A mapping module, used for mapping the global coordinates of the point cloud data in the point cloud data set to a two-dimensional plane to obtain a plurality of two-dimensional data; A classification module, used for classifying the plurality of two-dimensional data using a clustering algorithm to obtain a plurality of clusters; as well as The filtering module is used to filter the multiple clusters to obtain multiple target positions.

8. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images