An adaptive human 3D point cloud generation method based on 4D millimeter wave radar

Abstract

The application discloses a kind of self-adapting human 3D point cloud generation methods based on 4D millimeter wave radar, specifically includes: using millimeter wave radar to collect the original point cloud data of different activity posture of human body;The radar original point cloud data is preprocessed, and the target information of preliminary detection is obtained;According to the dynamic, static information in preliminary detection information, in combination with human body breathing and other characteristics, accurately locate each human body position, remove most multipath interference, obtain the human body data of rough estimate;Similar target is accumulated for multiple frames, and the target point cloud density is improved;Finally, the parameter adaptive DBSCAN algorithm designed further removes multipath noise, and obtains accurate human 3D point cloud data.The human posture point cloud generation method effectively improves the accuracy of generated point cloud while reducing the running time, has higher practicality and application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of millimeter wave radar human activity recognition, and particularly relates to a self-adaptive human 3D point cloud generation method based on 4D millimeter wave radar. BACKGROUND

[0002] In recent years, human activity recognition has made significant progress. Human activity recognition is a technology that uses artificial intelligence to analyze and process information such as human movement trajectories and postures, and to accurately recognize human activities. It is widely used in various fields. For example, in monitoring and video analysis, it can track and monitor abnormal behavior of people; in artificial intelligence technology and various human-computer interaction systems, it can recognize gestures and other information of users, improving user experience and convenience; in healthcare, it can provide real-time information for caregivers of patients and vulnerable groups. In addition, it also includes post-disaster search and rescue and intelligent service robots, etc.

[0003] According to different application scenarios, human activity recognition research can be mainly divided into two categories: computer vision-based and sensor-based human activity recognition methods. The computer vision-based method obtains video sequences or still images of human activities through a camera, and classifies the input data into basic activities by extracting features from the video or image. Computer vision-based human activity recognition has the advantages of high recognition accuracy and multiple recognition categories. Although computer vision-based human activity recognition has advantages, there are still many unsolved problems, such as cluttered background, partial occlusion, high environmental requirements, excessive cost, limited coverage, privacy and security issues, etc. Sensor-based human activity recognition systems mainly rely on sensors to collect human activity data, which can be divided into wearable and non-wearable types. Wearable devices monitor human acceleration, pulse, blood oxygen saturation, and other data to recognize human activities. However, wearable devices require users to actively wear them, and the user experience may not be convenient and comfortable. In recent years, a large number of non-contact sensors have been used to realize human activity recognition, such as WiFi, infrared sensors, etc. However, these sensors are sensitive to noise and environment, and are prone to false positives.

[0004] Compared with the above-mentioned sensors, the radar sensor based on radio frequency illuminates the target by electromagnetic wave, and expresses the real world by receiving the spatial heat map, point cloud map or distance Doppler map of the reflected or scattered RF intensity, effectively avoiding environmental interference. The millimeter wave radar sensor has the characteristics of high precision, low power consumption, compact structure, simple deployment and low cost, and can work in non-line-of-sight conditions, that is, it can penetrate some different kinds of thin layers, such as furniture, buildings and clothes, etc. In addition, since the millimeter wave radar is a non-intrusive technology, it will not cause privacy problems, etc. The above many advantages make the millimeter wave radar be widely used in security monitoring, medical diagnosis, attitude estimation, environmental monitoring and industrial automation fields.

[0005] With the continuous development of millimeter wave radar technology, it has also been widely applied in the field of human activity recognition. More and more researchers try to collect human 3D point cloud data by millimeter wave radar to realize human activity recognition. However, these studies mostly only focus on the feature processing after obtaining human activity 3D point cloud data, and do not pay much attention to the generation of human 3D point cloud. Due to the complexity of 3D point cloud data and the existence of noise interference, how to extract accurate and clean human 3D point cloud data from millimeter wave radar original point cloud data has become a very challenging problem.

[0006] To collect accurate human 3D point cloud data using millimeter wave radar, there are many challenges to be solved. First of all, in order to generate accurate and high-density human 3D point cloud, the millimeter wave radar needs to have high angular resolution and distance resolution. However, the number of antennas of commercial millimeter wave radar is usually limited, which makes the point cloud generated by each frame of receiving data too sparse to accurately reflect the human contour. Increasing the number of transceiver antennas can improve the angular resolution, but will lead to an increase in hardware cost. Secondly, it is also a great challenge to correctly associate each 3D point in the point cloud with the corresponding body segment. In some complex scenarios, the multi-path propagation caused by the large reflection of electromagnetic waves will cause the echo power to decay, producing phantom targets, so that the points in the environment are mistakenly considered as points from the human body. Finally, in the previous point cloud generation method, a large amount of manual labeling is usually needed to select the 3D point cloud belonging to the human body after the radar data is collected, which is not only a tedious work but also prone to human errors. SUMMARY

[0007] In order to solve the many shortcomings of the traditional point cloud generation method, this paper proposes a self-adaptive human 3D point cloud generation method based on 4D millimeter wave radar (Self-Adaption mPoint, SA-mPoint). It is a human 3D point cloud generation method that considers both the static and dynamic characteristics of radar reflection echoes and the breathing characteristics of the human body, and automatically finds the position of the human body without manual labeling.

[0008] In order to achieve the above object, the present application adopts the following technical scheme, comprising the following steps:

[0009] S1, radar data preprocessing: preprocessing the collected original millimeter wave radar human posture data to obtain an initial data set containing target information;

[0010] S2, rough detection of human target: according to the moving and static target information in the preprocessed data obtained in step S1, combining the human respiratory characteristics, accurately estimating the position of the existing human body, and obtaining the rough estimated human data;

[0011] S3, multi-frame dynamic accumulation: performing angle of arrival estimation on the rough estimated human data processed in S2 to obtain rough estimated human 3D point cloud data in polar coordinates, obtaining 3D point cloud data in Cartesian coordinate system through linear mapping, and performing multi-frame dynamic accumulation on similar targets to obtain dense rough estimated human 3D point cloud data;

[0012] S4, human 3D point cloud generation: clustering the rough estimated dense human 3D point cloud data generated in S3 to obtain accurate human 3D point cloud data.

[0013] Preferably, the preprocessing of the original millimeter wave radar human posture data in step S1 specifically comprises:

[0014] First, the original data is sequentially subjected to 1D, 2D-FFT to obtain range information (RI) and range-doppler information (R-DI), respectively. Then, the RI and R-DI are subjected to peak detection to obtain spatial frequency information of human information, and the target distance and velocity information can be obtained by formula and formula. Finally, the signal after non-coherent accumulation along the antenna of R-DI is subjected to CASO-CFAR detection to obtain an initial data set containing target information.

[0015]

[0016]

[0017] In the formula, f IF and Δw are the spatial frequency information of the human body, c is the speed of light, T c is the chirp pulse duration, λ is the wavelength, and T is the time interval between adjacent two chirp signals.

[0018] Preferably, the process of rough detection of human target according to the initial data set containing target in step S2 is:

[0019] Firstly, the distance gate number CurrBin corresponding to the point with the maximum signal-to-noise ratio of the moving target in the initial data set is found, and it is assumed that there is a human target at CurrBin. Then, the distance gate number RangeBin with the maximum RI echo power in the neighborhood of CurrBin is found, and whether there is a human respiratory feature at RangeBin is detected. Finally, whether there is a human target in the neighborhood of CurrBin is judged by comprehensively considering the respiratory feature and the signal-to-noise ratio of the moving target at CurrBin. When the RI detects a peak frequency of 0.1-0.5 Hz at RangeBin and the maximum signal-to-noise ratio of the point with the distance gate number CurrBin in the initial data set is greater than the average signal-to-noise ratio of the initial data set, it is indicated that there is a human target in the neighborhood of CurrBin. The above steps are repeated until there is no target meeting the condition.

[0020] The specific steps of the respiratory detection are as follows: firstly, the phase of the signal at RangeBin is extracted according to the formula Then, the phase is unwrapped, that is, when the phase difference between the continuous values is greater than π or less than -π, 2π is subtracted from the phase to obtain the real phase. Then, the unwrapped signal is phase-differentiated to enhance the respiratory signal. Finally, the obtained phase sequence is filtered through a low-pass filter to obtain the phase information in the detection range.

[0021]

[0022] In the formula, ImSignal is the imaginary part of the slow-time sequence of the initial data set at RangeBin, and ReSignal is the real part of the slow-time sequence of the initial data set at RangeBin.

[0023] Preferably, the specific process of the multi-frame dynamic accumulation in step S3 is as follows:

[0024] Firstly, the FFT is used to estimate the angle of arrival of the human data along the antenna dimension to obtain the spatial frequency information of the target. Then, the horizontal angle θ and the pitch angle of the target are calculated according to the formula The polar coordinate information of each target is obtained Then, the spatial coordinates (x, y, z) in the Cartesian coordinate system of each point are obtained according to the linear mapping. Finally, the multi-frame information is dynamically accumulated, specifically: after the end of each frame processing, the target data detected in the current frame and the previous frame are analyzed, and similar target data are merged, and the accumulation is stopped when the number of point clouds of the target with the least number is greater than a certain number.

[0025]

[0026] In the formula, d is the distance between the receiving antennas, Δw is the phase difference between the antennas, and λ is the wavelength of the transmitted electromagnetic wave.

[0027]

[0028] Preferably, the specific process of generating the human body 3D point cloud in step S4 is as follows:

[0029] Firstly, the spatial coordinate information P of n points of each target is obtained from the multi-frame dynamically accumulated data set i i i i i = 1, 2,..., n, and a Euclidean matrix Dist is constructed, which is defined as:

[0030]

[0031] In the formula, dist p＝2 (P i ,P j )i,j∈[1,n] represents the Euclidean distance between points P i and P j , which is defined as:

[0032]

[0033] Then, each row of the Dist matrix is traversed to obtain the maximum value maxD i i = 1, 2,..., n in each row, and a sequence DistMaxL = (maxD1, maxD2,..., maxD n ) is constructed. The minimum value DistMaxMin in DistMaxL is found, and the target point P k corresponding to the index k of the value is the human body center point. Define Epsilon as:

[0034]

[0035] In the formula, α is a scaling factor.

[0036] Next, the kth row of Dist is arranged from large to small, and the position inpt of the mutation point is found, and MinPts is defined as:

[0037]

[0038] In the formula, β is a scaling factor.

[0039] Finally, the multi-frame dynamically accumulated data is subjected to the DBSCAN algorithm with Epsilon and MinPts to generate the accurate human body 3D point cloud.

[0040] ​​​In summary, the application realizes accurate estimation of human activity point cloud by processing the data obtained by 4D millimeter wave radar, comprehensively considering echo dynamic and static information, combining target breathing characteristics, and greatly suppressing the influence of multipath noise through multi-frame dynamic accumulation and adaptive DBSCAN clustering algorithm.

[0041] The technical solutions of the application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 Radar signal processing flowchart;

[0043] Figure 2 Device and experimental site layout real scene diagram;

[0044] Figure 3 Each posture acquisition site diagram;

[0045] Figure 4 Each posture original point cloud diagram;

[0046] Figure 5 SA-mPoint each posture point cloud diagram;

[0047] Figure 6 TI-mPoint each posture point cloud diagram. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0049] In the description of the application, it should be understood that the terms “include” and “contain” indicate the existence of described features, whole, steps, operations, elements and / or components, but do not exclude the existence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0050] It should also be understood that the terms used in the specification of the application are only for the purpose of describing specific embodiments and do not intend to limit the application. As used in the specification and the appended claims of the application, the singular forms “a”, “an” and “the” are intended to include plural forms unless the context clearly indicates otherwise.

[0051] It should be further understood that the term "and / or" used in the description and claims of the application herein is used to mean any one and / or any combination of the associated listed items and includes all possible combinations, for example, A and / or B can mean only A, or only B, or both A and B. In addition, the character " / " in this document generally represents an "or" relationship between the front and rear associated objects.

[0052] It should be understood that, although the terms first, second, third, etc. can be used in embodiments of the application to describe a certain range, etc., these ranges should not be limited to these terms. These terms are only used to distinguish the ranges from each other. For example, the first preset range can also be referred to as the second preset range, and similarly, the second preset range can also be referred to as the first preset range without departing from the scope of the embodiments of the application.

[0053] Various structural diagrams according to the disclosed embodiments of the application are shown in the accompanying drawings. These drawings are not drawn to scale, in which certain details are exaggerated for the purpose of clarity and certain details can be omitted. The shapes of various regions, layers and their relative size and position relationship shown in the drawings are only exemplary, and in actuality, there can be deviations due to manufacturing tolerances or technical limitations, and a person skilled in the art can additionally design regions / layers with different shapes, sizes and relative positions according to actual needs.

[0054] The application provides a method for accurately generating human activity point cloud map using 4D millimeter wave radar, which uses commercial millimeter wave radar MMWCAS-RF-EVM and MMWCAS DSP-EVM to collect human activity data, and runs an instance through a convenient notebook computer configured as AMD Ryzen 75800H@3.2GHz with 16GB of memory. The detailed configuration parameters of the radar can be referred to Table 1. According to the configuration in the table, the millimeter wave radar can achieve a detection capability of a distance resolution of 0.0469m, a Doppler resolution of 0.0548m / s, a maximum measurement distance of 12m, and a maximum speed of 20.7813m / s.

[0055] Table 1 Radar configuration parameters

[0056]

[0057] The method processing flow can refer to Figure 1 , and the specific steps are as follows:

[0058] S1, radar data preprocessing: preprocessing the collected original millimeter wave radar human posture data to obtain an initial data set containing target information;

[0059] S2, human target rough detection: according to the moving and static target information in the preprocessed data obtained in step S1, the position of the human body is accurately estimated in combination with the human body breathing feature, and rough estimated human body data is obtained;

[0060] S3, multi-frame dynamic accumulation: the rough estimated human body data processed in S2 is subjected to angle of arrival estimation, rough estimated human body 3D point cloud data in polar coordinates is obtained, 3D point cloud data in Cartesian coordinates is obtained through linear mapping, and similar targets are subjected to multi-frame dynamic accumulation, and dense rough estimated human body 3D point cloud data is obtained.

[0061] S4, human body 3D point cloud generation: the dense rough estimated human body 3D point cloud data generated in S3 is subjected to clustering, and accurate human body 3D point cloud data is obtained.

[0062] The experimental environment is an empty classroom with a length of 10.7 m and a width of 7.6 m, and the on-site collection can refer to Figure 2 The radar is located on a tripod with a height of 1.1 m. All the original data collected by the millimeter wave radar board are transmitted to the computer through the MMWCAS-DSP-EVM, and the human activity 3D point cloud is generated by the computer running the SA-mPoint algorithm.

[0063] In order to discuss the advantages of SA-mPoint in generating human point cloud, the original data of an experimenter performing the actions of half-lifting arms, running, turning hands up and down, high-lifting arms, walking and two experimenters performing the actions of half-lifting arms and standing in the range of 1-1.6 m from the radar in the experimental environment shown in the previous section are collected, and the on-site pictures of each posture can refer to Figure 3 .

[0064] In order to measure the accuracy of human body 3D point cloud generated by different models, the present application defines A cc value representing the quality of the point cloud generated by the model. A cc is defined as:

[0065]

[0066] In the formula, Pn all represents the total number of point clouds generated by the model, and Pn in represents the number of correct point clouds. When the value of A cc is closer to 1, the accuracy of the point cloud generated by the model is higher.

[0067] In order to obtain the number of point clouds correctly estimated by the model, the position, width, height and depth information of the experimenter is recorded in detail before each data collection, and a point cloud evaluation frame is formed. The part of the point cloud generated by the model located in the frame is considered as the accurate human body 3D point cloud, and Pn inThe number of point clouds in the frame is the number of point clouds in the frame, and the point cloud evaluation frame data of each set of recorded poses can refer to Table 2.

[0068] Table 2 Point cloud evaluation window parameters

[0069]

[0070] The different human posture data collected above are processed by the present application and TI-mPoint respectively, and the results can be referred to Figures 4-6 . Among them, Figure 4 is the original point cloud graph, that is, the point cloud graph without any algorithm processing, and from the graph, it can be seen that there are multipath effects and noise caused by static targets. Figure 5 is the human posture point cloud graph generated by the present application, and the target is clearly classified and identified by the frame. From the graph, the human posture contour can be seen more obviously, and various noise interference is greatly suppressed, and the generated human point cloud graph has high quality. Figure 6 is the human posture point cloud graph generated by TI-mPoint, and the solid frame is the position of the human point cloud manually labeled later. It can be seen that although TI-mPoint can extract human posture information, it cannot accurately locate the precise position of the person, and needs to be manually labeled later, and there is serious noise interference caused by multipath effect, and the quality of the obtained human point cloud data is poor. Figure 5 and Figure 6 show that compared with TI-mPoint, the present application can generate more dense human point clouds, while greatly reducing the noise interference caused by multipath effect, and can also accurately locate the target position, and can also generate good point cloud quality for multiple targets.

[0071] In order to further compare the quality of the point clouds generated by the present application and TI-mPoint, the results of each posture data running by the two methods are respectively counted, and based on the above-mentioned indicators, the corresponding A cc values are calculated, and the specific indicators can be referred to Tables 3 and 4. Since TI-mPoint does not have the ability of target positioning and classification, the results of TI-mPoint in multiple target data processing are not listed. From Tables 3 and 4, in the single target experiment, the present application has an average point cloud generation accuracy of 97.94%, which is greatly improved compared with the average generation accuracy of 57.94% of TI-mPoint. In the multiple target experiment, the present application still has excellent performance, and for the experimenters standing and raising one arm, the human 3D point cloud generation accuracy is 93.3% and 97.1% respectively. According to the information in the table, compared with TI-mPoint, the number of point clouds generated by the present application is increased by an average of 87.94%, the accuracy is increased by an average of 78.3%, and the running time is reduced by an average of 11.41%.

[0072] Table 3 Running result statistics of each posture of the present application

[0073]

[0074] Table 4 TI-mPoint each attitude operation result statistics

[0075]

[0076] The above merely illustrates the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A 4D millimeter wave radar based adaptive human 3D point cloud generation method, characterized in that: The method comprises the following steps: S1, radar data preprocessing: preprocessing the collected original millimeter wave radar human posture data to obtain an initial data set containing target information; S2, rough human target detection: according to the dynamic and static target information in the preprocessed data obtained in step S1, combining the human breathing characteristics, accurately estimating the position of the existing human body, and obtaining the rough estimated human data; the process of roughly detecting the human target according to the initial data set containing the target is as follows: Firstly, find the distance gate number CurrBin corresponding to the point with the maximum signal-to-noise ratio of the dynamic target in the initial data set, and assume that there is a human target at CurrBin; then, find the distance gate number RangeBin with the maximum RI echo power in the neighborhood of CurrBin, and detect whether there is a human breathing feature at RangeBin; finally, consider the breathing feature and the signal-to-noise ratio of the dynamic target at CurrBin to determine whether there is a human target in the neighborhood of CurrBin; when the RI detects a peak frequency of 0.1-0.5 Hz at RangeBin and the maximum signal-to-noise ratio of the point with distance gate number CurrBin in the initial data set is greater than the average signal-to-noise ratio of the initial data set, it is indicated that there is a human target in the neighborhood of CurrBin; repeat the above steps until there is no target meeting the conditions; The specific steps of the respiratory feature detection are as follows: first, the phase of the signal at the RangeBin is extracted according to formula (1) Then, the phase is unwrapped, that is, when the phase difference between the continuous values is greater than π or less than -π, 2π is subtracted from the phase to obtain the true phase; then, the unwrapped signal is differentiated to enhance the respiratory signal; finally, the obtained phase sequence is filtered through a low-pass filter to obtain the phase information in the detection range. In the formula, ImSignal is the imaginary part of the slow time sequence of the initial data set at RangeBin, ReSignal is the real part of the slow time sequence of the initial data set at RangeBin, S3, multi-frame dynamic accumulation: performing angle of arrival estimation on the rough estimated human data processed in S2 to obtain rough estimated human 3D point cloud data in polar coordinates, obtaining 3D point cloud data in Cartesian coordinates through linear mapping, and performing multi-frame dynamic accumulation on similar targets to obtain dense rough estimated human 3D point cloud data; the specific process of multi-frame dynamic accumulation is as follows: First, the coarse estimated human data is processed by FFT to estimate the angle of arrival along the antenna dimension, and the spatial frequency information of the target is obtained. Then, the horizontal angle θ and the pitch angle φ of the target are calculated by formula (2) The polar coordinate information of each target is obtained Then, the spatial coordinates (x, y, z) of each point in the Cartesian coordinate system are obtained according to the linear mapping (3); finally, the dynamic accumulation of multiple frames of information is performed, specifically: after the end of each frame processing, the target data detected in the current frame and the previous frame are analyzed, similar target data are merged, and accumulation is stopped when the target with the least number of points is greater than a certain number of points; In the formula, d is the distance between the receiving antennas, Δw is the phase difference between the antennas, and λ is the wavelength of the transmitted electromagnetic wave, S4, human 3D point cloud generation: clustering the dense rough estimated human 3D point cloud data generated in S3 to obtain accurate human 3D point cloud data.

2. The adaptive human 3D point cloud generation method based on 4D millimeter wave radar according to claim 1, characterized in that, The preprocessing of the original millimeter wave radar human posture data in step S1 specifically comprises the following steps: Firstly, the original data is sequentially subjected to 1D and 2D-FFT to obtain range information (RI) and range-doppler information (R-DI), respectively; then, the RI and R-DI are subjected to peak value detection to obtain the spatial frequency information of the human information, and the target distance and speed information can be obtained according to formula (4) and formula (5); finally, the CASO-CFAR detection is performed on the non-coherent accumulated signal of R-DI along the antenna dimension to obtain the initial data set containing the target information; In the formula, f IF Δw is the spatial frequency information of the human body, c is the speed of light, T c is the chirp pulse duration, λ is the wavelength, and T is the time interval between adjacent two chirp signals.

3. The adaptive human 3D point cloud generation method based on 4D millimeter wave radar according to claim 1, characterized in that, The specific process of generating human 3D point cloud in step S4 is as follows: First, the spatial coordinate information P of n points of each target is obtained from the data set accumulated by S multi-frame dynamic i = (x i ,y i ,z i )i = 1, 2,..., n, and a Euclidean matrix Dist is constructed, which is defined as: where dist p＝2 (P i ,P j )i,j∈[1,n] denotes the Euclidean distance between points P i and P j , defined as: Then, traverse each row of the Dist matrix to get the maximum value maxD in each row i , i = 1, 2,..., n, construct a sequence DistMaxL = (maxD1, maxD2,..., maxDn) n ); find the minimum value DistMaxMin in DistMaxL, the index k corresponding to the target point P k of the value is the human center point; define Epsilon as: In the formula, α is a scaling factor; Then, the kth row of Dist is arranged from large to small, and the position of the mutation point inpt is found, and MinPts is defined as: In the formula, β is a scaling factor; Finally, the DBSCAN algorithm is performed on the data accumulated by multiple frames of dynamic data with Epsilon and MinPts to generate accurate human 3D point clouds.

Images