A Millimeter-Wave Radar Personnel Positioning Method Based on Cross-Supervised Learning

Abstract

A millimeter-wave radar personnel positioning method based on cross-supervised learning uses a binocular camera to collect depth point cloud data, and applies a human pose estimation algorithm to extract the spatial coordinate data of personnel targets from the depth point cloud data; uses the spatial coordinate data as labels and the echo data collected by the millimeter-wave radar as inputs to achieve cross-supervised learning training of the Hourgalss neural network; uses the millimeter-wave radar to collect radar echo data, inputs the radar echo data into the trained Hourglass neural network to predict the personnel coordinates based on the Hourglass neural network, then uses the AOA positioning algorithm to process the radar signals to calculate the personnel coordinates based on the AOA algorithm, and finally fuses the two methods to calculate the personnel coordinates to obtain the final personnel positioning result. The present invention can greatly improve the personnel positioning accuracy, thereby achieving the effect of having both the high-angle positioning accuracy of a commercial binocular camera and the high-distance positioning accuracy of a millimeter-wave radar only using the millimeter-wave radar.

Description

Technical Field

[0001] The present invention belongs to the field of radar positioning, and particularly relates to a millimeter-wave radar personnel positioning method based on cross-supervised learning. Background Art

[0002] Personnel positioning technology plays a very important role in today's economic society. For example, in large shopping malls, through real-time positioning of customers' positions, store information near the customers' positions or location information of relevant commodities can be provided to facilitate customers to quickly find the required commodities. It can also be used in places such as museums and exhibition halls to provide the best tour routes according to the different needs of visitors. Currently, the commonly used personnel positioning technologies mainly include Wi-Fi positioning, Bluetooth positioning, and infrared positioning. Millimeter-wave radar has higher positioning accuracy than other technologies due to its high bandwidth.

[0003] The millimeter-wave radar personnel positioning method obtains the distance and angle information of personnel by transmitting electromagnetic waves and receiving and analyzing the electromagnetic wave echoes, so as to calculate the coordinates of personnel in a specific coordinate system.

[0004] The current mainstream millimeter-wave radar personnel positioning methods are based on the AOA (Arrival of Angle) estimation algorithm to obtain the distance-angle map of the target, or use the BP (Back Projection) imaging algorithm to obtain the range-azimuth distance map of the target. The CFAR (Constant False-Alarm Rate) algorithm is used to determine the target point from the distance-angle map or the range-azimuth distance map, so as to obtain the distance and angle information of the personnel target. Then, through coordinate transformation, the coordinates of the personnel in a specific coordinate system can be calculated. Civilian small millimeter-wave radars such as the IWR6843 millimeter-wave radar produced by Texas Instruments can use a bandwidth of 4 GHz, and the range resolution can reach 4 cm. However, due to the limitation of the size, there are only 3 transmitting antennas and 4 receiving antennas, and the antenna aperture is small, resulting in an angular resolution of only 29°. The traditional millimeter-wave radar personnel positioning algorithm is affected by the low angular resolution of small civilian millimeter-wave radars, resulting in low positioning accuracy and unable to meet civilian needs. Summary of the Invention

[0005] Aiming at the defect that the existing millimeter-wave radar personnel positioning technology has low angle positioning accuracy in the application of millimeter-wave radar with small antenna aperture, the present invention combines the Hourglass convolutional neural network model widely used in engineering and proposes a millimeter-wave radar personnel positioning method based on cross-supervised learning. By integrating the traditional AOA positioning algorithm and combining the advantages of high range resolution of millimeter-wave radar to complement the range accuracy, the finally realized personnel positioning method not only gives full play to the high range resolution advantage of the millimeter-wave radar itself but also has the precise angle resolution of a commercial binocular camera under the condition of only using the millimeter-wave radar.

[0006] A millimeter-wave radar personnel positioning method based on cross-supervised learning includes the following steps:

[0007] Step 1, a binocular camera acquires the depth data matrix Depth and the RGB image matrix IMG of the area to be measured, and a millimeter-wave radar device concurrently acquires the echo data D0;

[0008] Step 2, use a human pose estimation algorithm to calculate the pixel coordinates of 18 human key points from IMG, and obtain the spatial Cartesian coordinates of 18 human key points from the depth image Depth according to the pixel coordinates and calculate the mean value (X, Y) as the personnel coordinates;

[0009] Step 3, use the AOA algorithm for the radar echo data D0 to obtain a range-angle map;

[0010] Step 4, use the personnel coordinates (X, Y) as labels and the range-angle map as input data to train the input Hourglass convolutional neural network to obtain an accurate radar echo model M;

[0011] Step 5, a millimeter-wave radar device acquires radar echo data D1, performs the AOA algorithm on the radar echo data D1 to obtain a range-angle map, inputs it into the radar echo model M, and predicts the Cartesian coordinates (X M , Y M ) of the personnel in the area to be measured, and perform coordinate transformation on (X M , Y M ) to obtain the personnel coordinates (R M , θ M ) in the radar polar coordinate system, where R M is the personnel distance based on Hourglass, and θ M is the personnel direction angle based on Hourglass;

[0012] Step 6: After denoising the range-angle diagram obtained in Step 5 using the OS-CFAR algorithm, use the DBSCAN clustering algorithm to cluster the personnel reflection signal points. After extracting the center (row, col) of each cluster, calculate the coordinates (R radar , θ radar ) of the personnel in the radar polar coordinate system through coordinate mapping;

[0013] Step 7: Use the KM weighted bipartite graph matching algorithm for (R radar , θ radar ) and (R M , θ M ), and output the final personnel coordinates (R' radar , θ' M ).

[0014] Furthermore, in Step 3, assume that the millimeter-wave radar device has a total of n virtual array elements, and the radar echo data D0[m] of the m-th virtual array element has nsamples sampling points. Use the AOA algorithm for D0: Concatenate the radar echo data of all n virtual array elements column by column to form a two-dimensional matrix with nsamples rows and n columns. Perform a fast Fourier transform (FFT) on the rows and columns of this two-dimensional matrix with nsamples sampling points respectively to obtain the range-angle diagram.

[0015] Furthermore, in Step 3, assume that the radar echo sampling rate is fs, the speed of light is c, the frequency modulation slope is K, the receiving antenna interval is d, and the wavelength is lambda. Then the value at the i-th row and j-th column in the range-angle diagram represents the echo power at a distance R and azimuth angle θ; where the distance R is:

[0016]

[0017] The azimuth angle θ is:

[0018]

[0019] Advantages of the present invention:

[0020] (1) Compared with traditional millimeter-wave radar personnel positioning algorithms, the present invention can also achieve high-angle-resolution positioning on millimeter-wave radar devices with small antenna apertures.

[0021] (2) Compared with the visual personnel positioning method that only uses binocular cameras, the present invention can achieve high-distance-resolution positioning, protect personnel privacy, and avoid the abuse of personnel image data.

[0022] (3) Compared with the positioning method that uses both a camera and a millimeter-wave radar simultaneously, the present invention can achieve a positioning accuracy similar to that of the positioning method using both a camera and a millimeter-wave radar by only using a millimeter-wave radar, reducing the complexity of the positioning method and saving costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the AOA positioning algorithm based on the millimeter-wave radar echo data of n virtual antenna elements in an embodiment of the present invention.

[0024] Figure 2 It is a pre-training flowchart of the Hourglass network based on cross-supervised learning in an embodiment of the present invention.

[0025] Figure 3 It is a flowchart of the millimeter-wave radar personnel positioning method based on the Hourglass network in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings of the specification.

[0027] The present invention is a millimeter-wave radar personnel positioning method based on cross-supervised learning. First, a binocular camera is used to obtain a depth data matrix Depth and an RGB image matrix IMG, and a millimeter-wave radar parallelly obtains radar echo data D0. A human pose estimation algorithm is used to calculate the pixel coordinates (x IMG , y IMG ) of 18 human key points of the person in the image. From this, the corresponding key point spatial coordinates (x D , y D ) are obtained from Depth, and the mean value of the 18 key point coordinates is taken as the person's coordinates (x, y).

[0028] Suppose the millimeter-wave radar device has a total of n virtual array elements, and the radar echo data D0[m] of the m-th virtual array element has nsamples sampling points. As Figure 1 shown, the AOA algorithm is used for D0: The radar echo data of all n virtual array elements are concatenated column by column into a two-dimensional matrix with nsamples rows and n columns. The FFT (Fast Fourier transform) of nsamples sampling points is performed on the rows and columns of this two-dimensional matrix respectively to obtain a range-angle diagram. Suppose the radar echo sampling rate is fs, the speed of light is c, the frequency modulation slope is K, the receiving antenna interval is d, and the wavelength is lambda. Then the value of the i-th row and j-th column in the range-angle diagram represents the reflected signal power intensity of the target at a distance of R and an azimuth angle of θ in the radar polar coordinate system. Among them, the distance R is:

[0029]

[0030] The azimuth angle θ is:

[0031]

[0032] Equations (1) and (2) complete the calculation from the coordinates of the range-angle diagram to the coordinates of the radar polar coordinate system. This process is called coordinate mapping.

[0033] Input the range-angle diagram into the Hourglass neural network, with the corresponding personnel coordinates (x, y) as labels. Continuously repeat the training process to adjust the interconnection relationship between the nodes of the neural network, making the Hourglass neural network closer to reality to obtain the radar echo model M. The training process is as Figure 2 shown. Among them, steps 1-4 are the pre-training process of cross-supervision learning of the Hourglass network, Figure 2 is the flowchart of steps 1-4, Figure 3 is the flowchart of steps 5-7.

[0034] After completing the training of the Hourglass neural network, in actual radar personnel positioning, the millimeter-wave radar obtains the echo data of the area to be measured. After using the AOA algorithm, a range-angle diagram is obtained. The range-angle diagram is used as the input of the radar echo model M to predict the Cartesian coordinates (X M , Y M ) of the personnel. Transform the Cartesian coordinates (X M , Y M )(The coordinates predicted by the Hourglass network inputting radar echo data are based on the binocular camera coordinate system because the labels used in the Hourglass network training are the coordinates based on binocular camera positioning) to the polar coordinates (R M , θ M ) based on the millimeter-wave radar coordinate system. R M is the distance of the personnel based on Hourglass, and θ MIt is the azimuth angle of the person based on the Hourglass. At the same time, the OS-CFAR algorithm is used to adaptively set the noise threshold for the range-angle diagram, remove the noise points in the diagram, and retain the effective reflection signal points of the person. Then, the DBSCAN algorithm is used to cluster the person's reflection signal points according to a certain distance weight, so that the reflection points from the same person are divided into one cluster. The center of the points in the same cluster is taken as the position (row, col) of the person in the range-angle diagram, where row and col represent the row index and column index of the person's center point in the range-angle diagram respectively. Under the conditions of knowing the radar signal sampling rate, signal frequency modulation slope, receiving antenna spacing distance, and signal wavelength, the spatial coordinates (R radar , θ radar ) of the person in the radar polar coordinate system are calculated according to (row, col) and formulas (1) and (2), where R radar is the distance of the person based on the AOA algorithm, and θ radar is the azimuth angle of the person based on the AOA algorithm.

[0035] Perform KM weighted bipartite graph matching on the point sets (R radar , θ radar ) and (R M , θ M ): Let (r radar , theta radar ) be a point in (R radar , θ radar ), and the Euclidean distance error threshold is dis. If the Euclidean distance d between a point (r M , θ M ) in (R M , θ M ) and (r radar , theta radar ) is less than dis, then it is considered that (r radar , theta radar ) and (rM, theta M ) are a set of successfully matched points, and the person's coordinates (r radar , theta M ) are output. Perform KM weighted bipartite graph matching on all points in (R radar , θ radar ) and (R M , θ M ), and output the final person coordinate point set (R′ radar , θ′ M ).

[0036] At present, the positioning accuracy of traditional millimeter-wave radar personnel positioning algorithms for personnel highly depends on the antenna aperture of millimeter-wave radar hardware. Small millimeter-wave radars do not have a large enough antenna aperture, making it difficult to accurately locate the position of personnel. The present invention has a low dependence on millimeter-wave radar hardware. Only by pre-training a radar echo model can high-precision personnel positioning be achieved.

[0037] The specific method process is as follows:

[0038] Steps 1-4 are the pre-training process of the Hourglass network based on cross-supervised learning, as Figure 2 shown.

[0039] Step 1: A binocular camera obtains the depth data matrix Depth and the RGB image matrix IMG of the area to be measured, and a millimeter-wave radar device simultaneously obtains the echo data D0.

[0040] Step 2: Use a human pose estimation algorithm to calculate the pixel coordinates of 18 human key points from IMG, and obtain the spatial Cartesian coordinates of 18 human key points from the depth image Depth according to the pixel coordinates and calculate the mean value (X, Y) as the personnel coordinates.

[0041] Step 3: Apply the AOA algorithm to the radar echo data D0 to obtain a range-angle map.

[0042] Step 4: Use the personnel coordinates (X, Y) as labels and the range-angle map as input data to train the input Hourglass convolutional neural network to obtain an accurate radar echo model M.

[0043] After completing the pre-training of the Hourglass network to obtain the model M, the following steps are carried out to complete millimeter-wave radar personnel positioning, as Figure 3 shown.

[0044] Step 5: A millimeter-wave radar device obtains radar echo data D1, applies the AOA algorithm to the radar echo data D1 to obtain a range-angle map, inputs it into the radar echo model M, and predicts the Cartesian coordinates (X M , Y M ) of the personnel in the area to be measured, and transforms (X M , Y M ) to obtain the personnel coordinates (R M , θ M ) in the radar polar coordinate system. R M is the personnel distance based on Hourglass, and θ M is the personnel direction angle based on Hourglass;

[0045] Step 6: After denoising the range-angle diagram obtained in Step 5 using the OS-CFAR algorithm, use the DBSCAN clustering algorithm to cluster the personnel reflection signal points. After extracting the center (row, col) of each cluster, calculate the coordinates (R radar , θ radar ) of the personnel in the radar polar coordinate system through coordinate mapping.

[0046] Step 7: Use the KM weighted bipartite graph matching algorithm for (R radar , θ radar ) and (R M , θ M ), and output the final coordinates (R' radar , θ' M ) of the personnel.

[0047] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modification or change made by those of ordinary skill in the art according to the disclosure of the present invention shall be included in the protection scope recorded in the claims.

Claims

1. A millimeter-wave radar personnel positioning method based on cross-supervised learning, characterized in that: Specifically, it includes the following steps: Step 1: The binocular camera obtains the depth data matrix Depth and the RGB image matrix IMG of the area to be measured, and the millimeter-wave radar device concurrently obtains the echo data D0; Step 2: Use the human pose estimation algorithm to calculate the pixel coordinates of 18 human key points from IMG, and obtain the spatial Cartesian coordinates of 18 human key points from the depth image Depth according to the pixel coordinates, and calculate the mean value (X, Y) as the personnel coordinates; Step 3: Apply the AOA algorithm to the radar echo data D0 to obtain the range-angle map; Step 4: Use the personnel coordinates (X, Y) as the label and the range-angle map as the input data to train the input Hourglass convolutional neural network to obtain an accurate radar echo model M; Step 5, the millimeter-wave radar device obtains radar echo data D1, performs the AOA algorithm on the radar echo data D1 to obtain a range-angle map, inputs the radar echo model M, and predicts the Cartesian coordinates (X M , Y M ) of the person in the area to be measured, and transforms the coordinates (X M , Y M ) to obtain the coordinates (R M , θ M ) of the person in the radar polar coordinate system. R M is the distance of the person based on Hourglass, and θ M is the direction angle of the person based on Hourglass; Step 6: After denoising the range-angle map obtained in Step 5 using the OS-CFAR algorithm, use the DBSCAN clustering algorithm to cluster the personnel reflection signal points. After extracting the center (row, col) of each cluster, calculate the coordinates (R radar , θ radar ) of the personnel in the radar polar coordinate system through coordinate mapping; Step 7, for (R radar , θ radar ), and (R M , θ M ), use the KM weighted bipartite graph matching algorithm to output the final personnel coordinates (R' radar , θ' M ).

2. The millimeter-wave radar personnel positioning method based on cross-supervised learning according to claim 1, wherein: In Step 3, assume that the millimeter-wave radar device has a total of n virtual array elements, and the radar echo data D0[m] of the m-th virtual array element has nsamples sampling points. Apply the AOA algorithm to D0: Concatenate the radar echo data of all n virtual array elements column by column to form a two-dimensional matrix with nsamples rows and n columns, and perform the fast Fourier transform FFT of nsamples sampling points on the rows and columns of this two-dimensional matrix respectively to obtain the range-angle map.

3. The millimeter-wave radar personnel positioning method based on cross-supervised learning according to claim 2, wherein: In Step 3, assume that the radar signal sampling rate is fs, the speed of light is c, the signal frequency modulation slope is K, the receiving antenna spacing distance is d, and the signal wavelength is lambda. The value at the i-th row and j-th column in the range-angle map represents the reflected signal power intensity of the target at a position with a distance of R and an azimuth angle of θ in the radar polar coordinate system; where the distance R is: The azimuth angle θ is:

Images