Detection method and device for key point of human body

Abstract

To provide a detection method and device for a key point of a human body.SOLUTION: A method includes the steps of: acquiring a radar signal by sensing an object using a radar; and acquiring a radar point group and zero Doppler FFT data on the basis of the radar signal; generating a virtual point group on the basis of the zero Doppler FFT data; acquiring a point group after merging by performing point group merging upon the radar point group and the virtual point group; and acquiring a key point by performing key point detection on the merged point group using a key point detection model. A key point of a human body is detected on the basis of the radar point group corresponding to a dynamic portion and the virtual point group corresponding to a static portion, such that more perfect target information can be provided. Thus, detection accuracy can be improved.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE The present invention relates to the technical field of radar detection. [Background technology]

[0002] The method for detecting human body keypoints based on radar point clouds has been widely applied because it does not violate privacy, can operate in dark or partially occluded scenes, and can obtain the real spatial positions of human body keypoints.

[0003] Figure 1 is a schematic diagram of the application of the human body keypoint detection method based on radar point cloud. As shown in Figure 1, when the human body is moving, the corresponding point cloud can be obtained from the radar, and the human body keypoint information can be detected from the point cloud, which can be used for motion detection, behavior analysis, fall detection, etc.

[0004] In motion detection and behavior analysis, the motion and behavior can be determined by analyzing the relative positions and statistical information of key points. In fall detection, it can be determined whether the target has fallen based on the height information of the key point positions of the human body.

[0005] It should be noted that the above description of the technical background is for the purpose of making the technical solutions of the present invention clear and complete, and is described for the purpose of making those skilled in the art understand. These technical solutions are merely described as part of the background technology of the present invention, and are not well known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0006] According to the discovery of the present inventor, the point cloud output by the radar (called radar point cloud or basic point cloud or original point cloud) is usually generated by the motion of the target and reflects the information of the moving part of the target, while the stationary part with zero velocity has no point cloud. Therefore, the point cloud may not be able to completely represent the target. In addition, the quality of the input radar point cloud directly affects the detection accuracy of the keypoints.

[0007] In view of the above technical problems, the embodiments of the present invention provide a method and apparatus for detecting keypoints, which can improve the detection accuracy. [Means for solving the problem]

[0008] In one aspect of an embodiment of the present invention, a method for detecting keypoints on a human body is provided, the method including the steps of: using a radar to sense an object and obtain a radar signal, and obtaining a radar point cloud and zero Doppler FFT data based on the radar signal; generating a virtual point cloud based on the zero Doppler FFT data; performing point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud; and performing keypoint detection on the merged point cloud using a keypoint detection model to obtain keypoints.

[0009] In another aspect of an embodiment of the present invention, there is provided an apparatus for detecting keypoints on a human body, the apparatus including: an acquisition unit that uses a radar to sense an object and acquire a radar signal, and acquires a radar point cloud and zero Doppler FFT data based on the radar signal; a generation unit that generates a virtual point cloud based on the zero Doppler FFT data; a merging unit that performs point cloud merging on the radar point cloud and the virtual point cloud and acquires a merged point cloud; and a detection unit that performs keypoint detection on the merged point cloud using a keypoint detection model, and acquires keypoints.

[0010] In another aspect of an embodiment of the present invention, there is provided an electronic device including a memory having a computer program stored therein and a processor, the processor being configured to execute the computer program to realize the above-mentioned method for detecting key points of a human body.

[0011] One of the advantageous effects of the embodiment of the present invention is as follows: by detecting the key points of the human body based on the radar point cloud corresponding to the moving parts and the virtual point cloud corresponding to the stationary parts, and taking the number of the radar point cloud plus the number of the virtual point cloud as the number of the merged point cloud, more complete target information can be provided, and the detection accuracy can be improved.

[0012] Certain embodiments of the invention are disclosed in detail, as set forth in the following description and in the drawings, which show how the principles of the invention may be employed, but are not intended to be limited in scope, and include all changes, modifications, and equivalents within the spirit and content of the appended claims. [Brief description of the drawings]

[0013] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the description of the text, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings. [Figure 1] 1 is a schematic diagram of the application of a human body keypoint detection method based on radar point cloud; [Diagram 2] FIG. 2 is a schematic diagram of an example of signal transmission and reception by a radar. [Diagram 3] FIG. 2 is a schematic diagram of an example of obtaining radar point cloud and zero Doppler FFT data based on a radar signal. [Figure 4] FIG. 2 is a schematic diagram of an example of zero Doppler FFT data. [Diagram 5] 1 is a schematic diagram of an example of a radar point cloud and Kinect detection results. [Figure 6] FIG. 1 is a schematic diagram of an example of true Kinect detection and predicted keypoint detection model values. [Figure 7] 2 is a schematic diagram of an example of a method for detecting key points of a human body according to an embodiment of the present invention; [Figure 8] 2 is a flowchart of an example of a method for detecting key points of a human body according to an embodiment of the present invention; [Figure 9] FIG. 13 is a schematic diagram of an example of generating a virtual point cloud based on zero Doppler FFT data. [Figure 10] FIG. 13 is a schematic diagram of an example of AM_difference(e_bin, a_bin, range_bin). [Figure 11] FIG. 2 is a schematic diagram of an example of radar point cloud features. [Figure 12] FIG. 2 is a schematic diagram of an example of a feature of a virtual point cloud. [Figure 13] FIG. 13 is a schematic diagram of an example of a merged point cloud. [Figure 14] FIG. 1 is a schematic diagram showing a comparison between the predicted values ​​of a model obtained by detecting keypoints using a method according to an embodiment of the present invention and the true values ​​of Kinet. [Figure 15] 1 is a schematic diagram of an example of a human body key point detection device according to an embodiment of the present invention; [Figure 16] 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The above and other features of the present invention will become apparent from the drawings and the following description. The specification and drawings disclose specific embodiments of the present invention, i.e., some embodiments according to the principles of the present invention. It is to be understood that the present invention is not limited to the described embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the appended claims.

[0015] In the embodiments of the present invention, the terms "first" and "second" are used to distinguish different elements by name, and do not imply any spatial arrangement or temporal order of these elements, and the elements are not limited to these terms. The term "and / or" includes any one or more of the listed terms and combinations thereof. The terms "including," "including," and "having" refer to the presence of stated features, elements, components or members, but do not exclude the presence or addition of one or more other features, elements, components or members.

[0016] In the present embodiment, the singular forms "a", "the", "the" and the like include the plural, meaning "a kind" or "a class", and are not limited to "one". Additionally, the term "said" includes both the singular and the plural, unless the context clearly dictates otherwise. Additionally, the term "according to" means "at least in part depending on" and the term "based on" means "at least in part based on", unless the context clearly dictates otherwise.

[0017] Features described and / or shown with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, may be combined with features in the other embodiments, or may be used in place of features of the other embodiments. The terms "including" or "having" refer to the presence of a described feature, element, component or member, but do not exclude the presence or addition of one or more other features, elements, components or members.

[0018] In the embodiment of the present invention, the radar may be a millimeter wave (mmWave) radar or a frequency modulated continuous wave (FMCW) radar, but the present invention is not limited thereto. FIG. 2 is a schematic diagram of an example of signal transmission and reception by a radar. As shown in FIG. 2, the radar emits an electromagnetic wave (transmission signal) through a transmission antenna, and receives a corresponding reflected wave (received signal, which may be referred to as radar echo information) after being reflected by different objects. By analyzing the radar echo information, information such as the position of the target from the radar, the radial moving speed, etc. can be effectively extracted, and these information can meet the needs of many application scenarios.

[0019] In an embodiment of the present invention, a radar signal can be obtained by using a radar to sense an object, and the radar signal is multi-dimensional information, such as distance, Doppler (velocity), angle, signal strength, etc. Based on the radar signal, a radar point cloud and zero-Doppler FFT (Fast Fourier Transform) data can be obtained.

[0020] Fig. 3 is a schematic diagram of an example of acquiring radar point cloud and zero-Doppler FFT data based on a radar signal. Fig. 4 is a schematic diagram of an example of zero-Doppler FFT data. As shown in Figs. 3 and 4, AD (analog / digital) conversion is performed on the radar signal, and distance FFT and Doppler FFT are performed on the converted signal to acquire Doppler FFT data, and as shown in Fig. 4, zero-Doppler FFT data (0-doppler-range FFT) can be extracted from the Doppler FFT data. In addition, CFAR (Constant False-Alarm Rate) is performed on the signal after the Doppler FFT, and angle FFT is performed on the obtained signal to acquire a radar point cloud.

[0021] In the above embodiment, the point cloud output by the radar is usually generated by the motion of the target and reflects the information of the moving part of the target, while the stationary part with zero velocity has no point cloud, so the point cloud may not completely represent the target.

[0022] FIG. 5 is a schematic diagram of an example of radar point clouds and Kinect detection results. As shown in FIG. 5(a), for a target sitting and waving, the radar point cloud reflects only the movement of its hands (i.e., moving parts). As shown in FIG. 5(b), for a target bending over to pick up an object, the radar point cloud reflects only the movement of its head and arms (i.e., moving parts). As shown in FIG. 5(c), for a target walking, the radar point cloud reflects only the movement of its hips and legs (i.e., moving parts). As shown in FIG. 5(d), for a target falling down or lying on its stomach, the radar point cloud reflects only the movement of its head and upper body (i.e., moving parts). As can be seen from the above, in the keypoint detection model, the quality of the input radar point cloud directly affects the detection accuracy of the keypoint. If the radar point cloud cannot completely represent the target, the detection of the keypoint may be erroneous.

[0023] Fig. 6 is a schematic diagram of an example of the true value of the Kinect detection and the predicted value of the keypoint detection model. As shown in Fig. 6(a), the predicted value 61 of the keypoint detection model and the true value 62 of the Kinect detection do not correspond at all. As shown in Fig. 6(b), when the predicted value 63 of the keypoint detection model is compared with the true value 64 of the Kinect detection, the predicted value 63 represents the target, but the position of the target represented by the predicted value 63 is above the true value 64 and is not accurate.

[0024] The following describes each aspect of the embodiment of the present invention with reference to the drawings. In the embodiment of the present invention, the detection target may be people of various ages, for example, elderly people, children, elderly people and / or nursing staff, children and / or guardians. The present invention is not limited thereto, and the detection target may be an animal with vital signs, or a robot without vital signs. The following describes an example in which the detection target is a human body.

[0025] <Example 1> An embodiment of the present invention provides a method for detecting key points of a human body. Figure 7 is a schematic diagram of an example of a method for detecting key points of a human body according to an embodiment of the present invention, and Figure 8 is a flowchart of an example of a method for detecting key points of a human body according to an embodiment of the present invention. As shown in Figure 7 and Figure 8, the method includes the following steps:

[0026] Step 701: Sense an object using a radar to obtain a radar signal, and obtain a radar point cloud and zero-Doppler FFT data based on the radar signal.

[0027] Step 702: Generate a virtual point cloud based on the zero Doppler FFT data.

[0028] Step 703: Perform point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud.

[0029] Step 704: Perform keypoint detection on the merged point cloud using a keypoint detection model to obtain keypoints.

[0030] Note that the above FIG. 7 merely illustrates an embodiment of the present invention, and the present invention is not limited thereto. For example, the execution order between each step may be adjusted as appropriate, other steps may be added, or some steps may be deleted. Those skilled in the art may make appropriate modifications based on the above content, and are not limited to the description of the above FIG. 7.

[0031] According to this embodiment, the radar point cloud and zero Doppler FFT data are obtained according to the radar signal, a virtual point cloud is generated according to the zero Doppler FFT data, and the radar point cloud and the virtual point cloud are merged, so that the key points of the human body are detected according to the radar point cloud corresponding to the moving part and the virtual point cloud corresponding to the stationary part, thereby providing more complete target information, and improving the detection accuracy.

[0032] The embodiment of the present invention is not limited to a method of acquiring radar point cloud and zero Doppler FFT data.

[0033] For example, the radar point cloud may be acquired according to the process shown in Fig. 3. That is, AD conversion is performed on the radar signal, distance FFT is performed on the AD converted radar signal, Doppler FFT is performed on the radar signal on which the distance FFT has been performed, CFAR (constant probability of false alarm detection) is performed on the radar signal on which the Doppler FFT has been performed, and angle FFT is performed on the radar signal on which the CFAR has been performed, to acquire the radar point cloud. The relevant art may be referred to for specific implementation procedures, and description thereof will be omitted here.

[0034] Also, for example, zero Doppler FFT data may be acquired according to the process shown in Fig. 3 and the process shown in Fig. 4. That is, AD conversion is performed on the radar signal, distance FFT is performed on the AD converted radar signal, and Doppler FFT is performed on the radar signal that has been subjected to the distance FFT to acquire Doppler FFT data, and zero Doppler FFT data is extracted from the Doppler FFT data. For specific implementation procedures, reference may be made to related art, and description thereof will be omitted here.

[0035] In some embodiments, the virtual point cloud is obtained from the zero-Doppler FFT data, for example, the virtual point cloud may be obtained from the zero-Doppler FFT data according to the method shown in Figure 9. As shown in Figure 9, the method includes the following steps:

[0036] Step 910: A two-dimensional Fourier transform is performed on the zero-Doppler FFT data (0-doppler-range-FFT(ant_cnt, range_bin)) to obtain angle FFT data (angle-FFT(e_bin, a_bin, range_bin)).

[0037] Step 920: Based on the angle FFT data of the T frames, the average value of the difference between the immediately preceding frame and the immediately following frame (AM_difference(e_bin, a_bin, range_bin)) is calculated.

[0038] Step 930: CFAR (Constant False Alarm Rate Detection) is performed on the three directions of the average value, namely the vertical angle, horizontal angle, and distance, and the position (e, a, r) ​​whose values ​​are all greater than a predetermined CFAR threshold is determined to be the target position of the virtual point cloud.

[0039] Step 940: Transform the target position (e, a, r) ​​into coordinates (x, y, z) to obtain the virtual point cloud.

[0040] In the above embodiment, the zero-Doppler FFT data may be expressed as 0-doppler-range-FFT(ant_cnt, range_bin), where ant_cnt is the number of antenna pairs and range_bin is the number of range Fourier transformed frequency points, and a two-dimensional Fourier transform is performed on 0-doppler-range-FFT(ant_cnt, range_bin) to obtain angle FFT data.

[0041] In the above embodiment, the angle FFT data may be expressed as angle-FFT(e_bin, a_bin, range_bin), where e_bin and a_bin are vertical and horizontal angular frequency points, respectively, and the meaning of range_bin is the same as that described above, that is, the frequency point obtained by the range Fourier transform. The meaning of angle FFT may be referred to in the related art, and the description is omitted here.

[0042] In the above embodiment, based on the angle FFT data of T consecutive frames, the average value of the difference value of the angle FFT data between the previous frame and the next frame may be calculated and expressed as AM_difference(e_bin, a_bin, range_bin). AM_difference(e_bin, a_bin, range_bin) is the average value of the difference of the amplitude values ​​at the same distance frequency point, horizontal angle frequency point, and vertical angle frequency point in a certain time. Here, T is a natural number greater than 0, and the present invention is not limited to the value of T.

[0043] In the above embodiment, a certain false alarm probability detection is performed for each of the three directions of AM_difference (e_bin, a_bin, range_bin), namely, vertical angle, horizontal angle, and distance. If the values ​​of the position (e, a, r) ​​are all greater than a predetermined CFAR threshold, the position is determined to be the target position of the virtual point cloud, and the target position (e, a, r) ​​is converted to coordinates (x, y, z) to obtain the coordinates of the virtual point cloud.

[0044] FIG. 10 is a schematic diagram of an example of AM_difference(e_bin, a_bin, range_bin). As shown in FIG. 10, in AM_difference(e_bin, a_bin, range_bin), e_bin=32, a_bin=32, and range_bin=128. According to the method of the above embodiment, constant false alarm probability detection is performed on AM_difference(e_bin, a_bin, range_bin) from three directions, and (e j ,a k ,r i If the value of (e) is greater than the pre-set CFAR threshold, j ,a k ,r i ) is determined as the target position, and the position (e j ,a k ,r i ) into coordinates (x, y, z) and obtain the coordinates (x, y, z) of the virtual point cloud to obtain the virtual point cloud.

[0045] In the example of FIG. 10, only one target position is described as an example, but the present invention is not limited to this. All positions whose values ​​are greater than a pre-set CFAR threshold may be determined as target positions, and the corresponding coordinates may be set as the coordinates of the virtual point cloud.

[0046] In the above embodiment, the CFAR threshold value is not limited to the value, and may be set according to experience or need. Also, different CFAR threshold values ​​may be set for different targets, such as different people. Alternatively, different CFAR threshold values ​​may be set for different radars.

[0047] The above embodiment is not limited to the method of constant false alarm probability detection, for example, the method of unit average detection (CA-CFAR) or the method of minimum selection constant false alarm probability detection (SO-CFAR) may be adopted, but the present invention is not limited thereto.

[0048] For example, in the CA-CFAR method, N reference units are set on both sides of the detection target unit, and the average value of a total of 2N reference units is calculated as an estimate of the background clutter power. Also, for example, in the SO-CFAR method, average values ​​X and Y are calculated for the reference units, and the one of X and Y with the smaller average value is selected as the estimate of the background clutter power of the detection target unit D. The principles of CA-CFAR and SO-CFAR may be referred to in the related art, and will not be described here.

[0049] In the above embodiment, the constant false alarm probability detection in different directions can adopt the same method or different methods. Still take Fig. 10 as an example, for example, the constant false alarm probability detection is performed in the range direction using SO-CFAR, and the constant false alarm probability detection is performed in the horizontal angle and vertical angle directions using CA-CFAR. The following table shows the CFAR threshold corresponding to the above example.

[0050] [Table 1] In the above embodiment, the zero Doppler FFT data may be the zero Doppler FFT data in the entire distance range, or the zero Doppler FFT data in a certain distance range, because the data in a too far distance is unlikely to correspond to the detection target, that is, the data in this part is almost meaningless for detecting the target keypoints.

[0051] In one example, the range_bin in the above zero Doppler FFT data may be determined according to the following method.

[0052] S1: Based on the original radar point cloud information, obtain a set of distance-frequency points where the original radar point cloud is located, and calculate R0 = {r i , i≦N}, where N is the number of original radar point clouds.

[0053] S2:r i Within the vicinity of the distance frequency point range r for generating a virtual point cloud i Select _virtual and press r i _virtual=(r i -r limit ,r i +r limit ), where r limit =Nr, for example, Nr=10, where Nr is the number of distance frequency points, and Nr=10 represents a range of 10 distance frequency points before and after the distance frequency point occupied by the point group of interest, i.e., a range of 20 distance frequency points.

[0054] S3: Finally, the distance frequency point set for generating the virtual point cloud, i.e., all r i Calculate the union of _virtual and set the union as range_bin.

[0055] The above example determination methods are merely illustrative, and range_bin may be determined in other ways.

[0056] In the above embodiment, preferably, as shown in FIG. 9, the method may further include the following steps:

[0057] Step 950: Filter the virtual point cloud based on the distance between the virtual point cloud and the radar point cloud, such that virtual point clouds outside a predetermined distance range are filtered out.

[0058] For example, virtual point clouds whose distance from the center of the radar point cloud is equal to or greater than a first predetermined threshold d1 are filtered out, i.e., the distance between the virtual point cloud and the center of the radar point cloud is less than d1, e.g., less than 1.2 meters.

[0059] Also, for example, virtual point clouds whose chamfer distance from the radar point cloud is equal to or greater than a second predetermined threshold d2 are removed by filtering, i.e., the chamfer distance between the radar point cloud and the virtual point cloud is less than d2, for example, less than 1 meter.

[0060] The present invention is not limited to the values ​​of d1 and d2, and may be set according to the type of detection target and / or the type and / or experience of the radar, and may be the same or different. This makes it possible to reduce the amount of calculation in the subsequent keypoint detection process by filtering out virtual point clouds that are unlikely to correspond to the detection target.

[0061] In the above embodiment, as shown in FIG. 9, the method may further include the following steps:

[0062] Step 960: Reorder the virtual point cloud according to a predetermined rule.

[0063] Step 970: Select a predetermined number of the top sorted virtual point clouds as the final result.

[0064] In some examples, the predetermined rule includes an average value of difference values ​​between the immediately preceding and subsequent frames of the angular FFT data of the T frames, i.e., sorting the virtual point clouds according to the average value of difference values ​​between the immediately preceding and subsequent frames of the angular FFT data of the T frames, and selecting, for example, the top M virtual point clouds as supplementary points of the radar point cloud.

[0065] In some other examples, the predetermined rule includes the distance between the virtual point cloud and the center of the radar point cloud, i.e., sorting the virtual point clouds according to the distance between the virtual point cloud and the center of the radar point cloud, for example, selecting the M virtual point clouds with the smallest distance.

[0066] In the above example, M is a natural number greater than 0, but the present invention is not limited to the value of M.

[0067] Note that the above FIG. 9 merely illustrates an embodiment of the present invention, and the present invention is not limited thereto. For example, the execution order between each step may be adjusted as appropriate, other steps may be added, or some steps may be deleted. Those skilled in the art may make appropriate modifications based on the above content, and are not limited to the description of the above FIG. 9.

[0068] In the above embodiment, the radar point cloud and the virtual point cloud are merged, i.e., the features and numbers are merged. In some examples, the sum of the number of radar point clouds and the number of virtual point clouds is the number of merged point clouds, and the features of the radar point cloud and the features of the virtual point cloud are each expanded to the same number (digits) to obtain the merged point cloud.

[0069] Currently, there are six features of the radar point cloud: x, y, z, v, p, and fID, where x, y, and z are the coordinates of the spatial location of the radar point cloud, v is the Doppler, p is the signal strength, and fID is the frame number. Figure 11 is a schematic diagram of an example of the features of the radar point cloud.

[0070] In the above example, AM_difference (i.e., the average value of the difference between the immediately preceding frame and the immediately following frame in the angular FFT data of the T frames) may be set as the seventh feature of the radar point cloud. Here, among the seven features of the radar point cloud, the values ​​of x, y, z, v, p, and fID are true values, i.e., true values ​​obtained based on the radar signal, and the value of AM_difference may be set to a fixed value, for example, 0, or may be set to the true value of the corresponding virtual point cloud.

[0071] In the above example, the virtual point cloud has the same seven features as the radar point cloud. Of the seven features of the virtual point cloud, the values ​​of v and p may be set to fixed values, e.g., 0, the values ​​of x, y, and z may be set to true values, e.g., the coordinates of the virtual point cloud obtained by step 902, the value of fID may be a fixed value, e.g., a constant C, e.g., C=0 or 1 or T+1, etc., where T may be the number of frames for calculating AM_difference, and the value of AM_difference may be a fixed value, e.g., 0, or a true value, e.g., the average value of the difference values ​​obtained by step 920. FIG. 12 is a schematic diagram of an example of the features of the virtual point cloud.

[0072] According to the above embodiment, the point cloud is expanded from the basic point cloud [N, 6] to [N+M, 7], where N is the number of radar point clouds, M is the number of virtual point clouds, and 6 and 7 are the numbers of features respectively, that is, the features of the point cloud are expanded from 6 to 7. Figure 13 is a schematic diagram of an example of a merged point cloud.

[0073] In an embodiment of the present invention, after the merged point cloud is obtained, i.e., after the basic point cloud (the radar point cloud obtained based on the radar signal) is expanded by the virtual point cloud, keypoint detection may be performed on the merged point cloud using a keypoint detection model to obtain keypoints of the detection target (e.g., a human body).

[0074] The embodiments of the present invention are not limited to the configuration and implementation method of the keypoint detection model, and specific reference may be made to related art.

[0075] The following takes several operation categories as examples to illustrate the performance of the method according to the embodiment of the present invention.

[0076] In the following description, the following nine action categories are taken as an example. Action 21 is a walk-stand action. Action 22 is a stand-sit action. Action 23 is a walk-lie bed action. Action 24 is a walk-lie floor action. Action 25 is a sit-lie floor action. Action 26 is a lie bed-lie floor action. Action 27 is a stand-lie floor action. Action 28 is a walk-fall facing radar action. Action 29 is a walk-fall far from radar action.

[0077] The following table compares the mean absolute error (MAE) obtained from the method according to the embodiment of the present invention with the MAR obtained from the conventional method.

[0078] [Table 2] As can be seen from the above table, according to an embodiment of the present invention, assuming that 30 virtual point clouds are selected and merged based on the zero Doppler FFT data of 10 consecutive frames and the amplitude difference of the angle FFT data, compared to the baseline, the mean absolute error (MAE) is reduced and the detection accuracy is improved for each motion category, regardless of whether the 7th and 6th are 0 and 1 or 0 and 11, respectively.

[0079] Fig. 14 is a schematic diagram of the prediction value of the model obtained by detecting keypoints by the method according to the embodiment of the present invention versus the true value of Kinet. In the example of Fig. 14, the light dots represent the radar point cloud, and the dark triangles represent the virtual point cloud.

[0080] As shown in (a) of FIG. 14, for the falling motion, the model predicted value 142 (right) obtained by the method according to the embodiment of the present invention is closer to the Kinet true value than the model predicted value 141 (left) obtained by the conventional method. As shown in (b) of FIG. 14, for the standing motion, the model predicted value 144 (right) obtained by the method according to the embodiment of the present invention is closer to the Kinet true value than the model predicted value 143 (left) obtained by the conventional method. As shown in (c) of FIG. 14, for the bending motion, the model predicted value 146 (right) obtained by the method according to the embodiment of the present invention is closer to the Kinet true value than the model predicted value 145 (left) obtained by the conventional method.

[0081] The above-mentioned embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto, and appropriate modifications may be made based on the above-mentioned various embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

[0082] According to this embodiment, the virtual point cloud can detect key points of the human body based on the radar point cloud corresponding to the moving parts and the virtual point cloud corresponding to the stationary parts, thereby providing more complete target information and improving detection accuracy.

[0083] <Example 2> The embodiment of the present invention provides a human body key point detection device, and the problem solving principle is the same as that of the first embodiment, so the description of the similar contents will be omitted.

[0084] 15 is a schematic diagram of an example of a human body key point detection device 1500 according to an embodiment of the present invention. As shown in FIG. 15, the human body key point detection device 1500 according to an embodiment of the present invention includes the following units:

[0085] The acquisition unit 1501 senses an object using a radar to acquire a radar signal, and acquires a radar point cloud and zero Doppler FFT data based on the radar signal.

[0086] The generator 1502 generates a virtual point cloud based on the zero Doppler FFT data.

[0087] A merging unit 1503 performs point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud.

[0088] The detection unit 1504 performs keypoint detection on the merged point cloud using a keypoint detection model, and obtains keypoints.

[0089] In some embodiments, the acquisition unit 1501 acquires the zero Doppler FFT data according to the following method.

[0090] S1: Perform AD conversion on the radar signal.

[0091] S2: Perform distance FFT on the radar signal after AD conversion.

[0092] S3: A Doppler FFT is performed on the radar signal that has been subjected to the range FFT, and the zero Doppler FFT data is obtained.

[0093] In some embodiments, the acquisition unit 1501 acquires the radar signal according to the following method.

[0094] S1: Perform AD conversion on the radar signal.

[0095] S2: Perform distance FFT on the radar signal after AD conversion.

[0096] S3: A Doppler FFT is performed on the radar signal that has undergone range FFT.

[0097] S4: CFAR (Constant False Alarm Probability Detection) is performed on the radar signal that has been subjected to the Doppler FFT.

[0098] S5: Perform angular FFT on the CFARed radar signal to obtain the radar point cloud.

[0099] In some embodiments, the generator 1502 performs the following steps in generating a virtual point cloud based on the zero-Doppler FFT data.

[0100] S1: A two-dimensional Fourier transform is performed on the zero-Doppler FFT data (0-doppler-range-FFT(ant_cnt, range_bin)) to obtain angle FFT data (angle-FFT(e_bin, a_bin, range_bin)).

[0101] S2: Based on the angle FFT data of the T frames, calculate the average value (AM_difference(e_bin, a_bin, range_bin)) of the difference values ​​between the immediately preceding frame and the immediately following frame.

[0102] S3: CFAR (Constant False Alarm Rate Detection) is performed on the three directions of the average value, namely the vertical angle, horizontal angle, and distance, and the position (e, a, r) ​​whose values ​​are all greater than a predetermined CFAR threshold is determined to be the target position of the virtual point cloud.

[0103] S4: Convert the target position into coordinates to obtain the virtual point cloud.

[0104] In some embodiments, as shown in FIG. 15, the apparatus 1500 for detecting key points of a human body further includes the following units:

[0105] The filtering unit 1505 filters the virtual point cloud based on the distance between the virtual point cloud and the radar point cloud so as to remove the virtual point cloud outside a predetermined distance range by filtering.

[0106] In the above embodiment, the filtering unit 1505 may filter out virtual point clouds whose distance from the center of the radar point cloud is equal to or greater than a first predetermined threshold d1, or may filter out virtual point clouds whose chamfer distance from the radar point cloud is equal to or greater than a second predetermined threshold d2.

[0107] In some embodiments, as shown in FIG. 15, the apparatus 1500 for detecting key points of a human body further includes the following units:

[0108] The rearrangement unit 1506 rearranges the virtual point group according to a predetermined rule.

[0109] The selection unit 1507 selects a predetermined number of the top sorted virtual point groups as a final result.

[0110] In the above embodiment, the predetermined rule may include an average value of difference values ​​between a frame immediately before and a frame immediately after the angle FFT data of the T frames. That is, the sorting unit 1506 sorts the virtual point groups according to the average value of difference values ​​between a frame immediately before and a frame immediately after the angle FFT data of the T frames, and the selection unit 1507 selects the top M virtual point groups.

[0111] In the above embodiment, the predetermined rule may include the distance between the virtual point cloud and the center of the radar point cloud. That is, the sorting unit 1506 sorts the virtual point cloud according to the distance between the virtual point cloud and the center of the radar point cloud, and the selection unit 1507 selects M virtual point clouds with the smallest distance.

[0112] In some embodiments, the merging unit 1503 performs the following steps when performing point cloud merging on the radar point cloud and the virtual point cloud.

[0113] S1: The sum of the number of radar point clouds and the number of virtual point clouds is the number of merged point clouds.

[0114] S2: Expand the radar point cloud features and the virtual point cloud features to seven, corresponding to v, p, x, y, z, fID and AM_difference, respectively.

[0115] Here, for the radar point cloud features, the values ​​of v, p, x, y, z, and fID are true values, and the value of AM_difference is 0 or the true value of the corresponding virtual point cloud.

[0116] For the features of the virtual point cloud, the values ​​of v and p are 0, the values ​​of x, y, and z are true values, the value of fID is a fixed value, and the value of AM_difference is 0 or a true value.

[0117] It should be noted that the above merely describes each component or module related to the present invention, and the present invention is not limited thereto. The above-mentioned human body key point detection device 1500 may include other components or modules, and the specific contents of these components or modules may refer to the related art.

[0118] For simplicity, Fig. 15 merely illustrates the connection relationships or signal directions between each component or module as an example, and it is clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned various components or modules may be realized by hardware devices such as a processor and a memory, and the embodiment of the present invention is not limited thereto.

[0119] The above-mentioned embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto, and appropriate modifications may be made based on the above-mentioned various embodiments. For example, the above-mentioned embodiments may be used alone, or one or more of the above-mentioned embodiments may be used in combination.

[0120] According to this embodiment, by detecting key points of the human body based on the radar point cloud corresponding to the moving parts and the virtual point cloud corresponding to the stationary parts, more complete target information can be provided, thereby improving the detection accuracy.

[0121] <Example 3> An embodiment of the present invention provides an electronic device including the human body key point detection device 1500 described in the second embodiment, the contents of which are incorporated herein by reference. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smartphone, etc., but the embodiment of the present invention is not limited thereto.

[0122] Fig. 16 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in Fig. 16, the electronic device 1600 according to the embodiment of the present invention includes a processor (e.g., a central processing unit (CPU)) 1610 and a memory 1620. The memory 1620 is connected to the processor 1610. The memory 1620 may store various data and may further store an information processing program 1621. The program 1621 is executed under the control of the processor 1610.

[0123] In some aspects, the function of the human body keypoint detection device 1500 described in Example 2 may be integrated into the processor 1610. Here, the processor 1610 may be configured to realize the method described in Example 1.

[0124] In some aspects, the human body keypoint detection apparatus 1500 described in Example 2 may be respectively configured with a processor 1610, for example, the human body keypoint detection apparatus 1500 described in Example 2 may be a chip connected to the processor 1610 and configured to realize the functions of the human body keypoint detection apparatus 1500 described in Example 3 under the control of the processor 1610.

[0125] 16, the electronic device 1600 may further include an input / output (I / O) device 1630 and a display 1640. Here, the functions of the components are similar to those of the conventional technology, and the description thereof will be omitted here. Note that the electronic device 1600 does not need to include all the components shown in FIG. 16. The electronic device 1600 may also include components not shown in FIG. 16, and may refer to the conventional technology.

[0126] An embodiment of the present invention provides a computer-readable program that, when executed in an electronic device, causes a computer to execute the method described in embodiment 1 in the electronic device.

[0127] An embodiment of the present invention further provides a storage medium having stored thereon a computer-readable program for causing a computer to execute the method according to embodiment 1 in an electronic device.

[0128] The above-mentioned apparatus and method of the present invention may be realized by hardware or a combination of hardware and software. The present invention relates to a computer-readable program, which, when executed by a logic unit, causes the logic unit to realize the above-mentioned apparatus or components, or to realize the above-mentioned various methods or steps. The present invention also relates to a storage medium for storing the above-mentioned program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0129] The methods / apparatuses described with reference to the embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in the drawings, or one or more combinations of the functional block diagrams, may correspond to software modules of a computer program process or hardware modules. These software modules may correspond to steps shown in the drawings. These hardware modules may be implemented by implementing these software modules in hardware, for example, using a field programmable gate array (FPGA).

[0130] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from the storage medium and writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal, or in a memory card inserted in the mobile terminal. For example, if the device (e.g., a mobile terminal) uses a MEGA-SIM card or a large capacity flash memory device with a relatively large capacity, the software module may be stored in the MEGA-SIM card or the large capacity flash memory device.

[0131] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the drawings may be implemented in a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the drawings may be implemented in, for example, a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0132] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

[0133] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above examples. (Appendix 1) A method for detecting key points on a human body, comprising: S1: detecting an object using a radar and acquiring a radar signal; S2: obtaining radar point cloud and zero Doppler FFT data based on the radar signal; S3: generating a virtual point cloud based on the zero Doppler FFT data; S4: performing point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud; S5: performing keypoint detection on the merged point cloud using a keypoint detection model to obtain keypoints. (Appendix 2) In S2, the zero Doppler FFT data is performing AD conversion on the radar signal; After AD conversion, the radar signal is subjected to distance FFT. 2. The method described in Appendix 1, obtained by performing a Doppler FFT on a radar signal that has been subjected to a range FFT and obtaining the zero Doppler FFT data. (Appendix 3) In the S2, the radar point cloud is performing AD conversion on the radar signal; After AD conversion, the radar signal is subjected to distance FFT. A Doppler FFT is performed on the radar signal that has undergone range FFT. CFAR (Constant False Alarm Probability Detection) is performed on the radar signal that has undergone Doppler FFT. 2. The method of claim 1, obtained by performing an angular FFT on a CFARed radar signal to obtain the radar point cloud. (Appendix 4) The S3 is S31: performing a two-dimensional Fourier transform on the zero-Doppler FFT data (0-doppler-range-FFT(ant_cnt, range_bin)) to obtain angle FFT data (angle-FFT(e_bin, a_bin, range_bin)); S32: Calculating an average value (AM_difference(e_bin, a_bin, range_bin)) of the difference between the immediately preceding frame and the immediately following frame based on the angle FFT data of T frames, where T is greater than 0; S33: A step of performing CFAR (constant false alarm probability detection) on the three directions of the vertical angle, horizontal angle, and distance of the average value, and determining a position where the value of position (e, a, r) ​​is greater than a predetermined CFAR threshold as a target position of the virtual point cloud; S34: converting the target positions into coordinates to obtain the virtual point cloud. (Appendix 5) 5. The method of claim 4, wherein the constant false alarm probability detection includes unit average detection (CA-CFAR) or minimum selection constant false alarm probability detection (SO-CFAR). (Appendix 6) S35: The method of claim 4, further comprising a step of filtering the virtual point cloud based on a distance between the virtual point cloud and the radar point cloud to filter out virtual point clouds outside a predetermined distance range. (Appendix 7) Filtering out virtual points whose distance from the center of the radar point cloud is equal to or greater than a first predetermined threshold value d1; or 7. The method of claim 6, further comprising filtering out virtual point clouds whose chamfer distance from the radar point cloud is greater than or equal to a second predetermined threshold d2. (Appendix 8) S36: rearranging the virtual point cloud according to a predetermined rule; S37: The method of claim 4, further comprising the step of selecting a predetermined number of the top sorted virtual point clouds as a final result. (Appendix 9) the predetermined rule includes an average value of difference values ​​between a frame immediately before and a frame immediately after the T frames of angular FFT data; 9. The method according to claim 8, wherein the virtual point clouds are sorted according to an average value of difference values ​​between a previous frame and a next frame of the angular FFT data of T frames, and a top M virtual point clouds are selected, where M is greater than 0. (Appendix 10) the predetermined rule includes a distance between the virtual point cloud and a center of the radar point cloud; 9. The method of claim 8, wherein the virtual point clouds are sorted according to a distance between the virtual point cloud and a center of the radar point cloud, and M virtual point clouds with the smallest distances are selected, where M is greater than 0. (Appendix 11) The above S4 is A step of setting the sum of the number of the radar point clouds and the number of the virtual point clouds as the number of the merged point clouds; augmenting the radar point cloud features and the virtual point cloud features into seven features corresponding to v, p, x, y, z, fID and AM_difference, respectively; For the radar point cloud features, the values ​​of v, p, x, y, z, and fID are true values, and the value of AM_difference is 0 or a true value of the corresponding virtual point cloud; For the virtual point group features, the values ​​of v and p are 0, the values ​​of x, y, and z are true values, the value of fID is a fixed value, and the value of AM_difference is 0 or a true value; The method described in Appendix 1, wherein v is Doppler, p is signal strength, fID is frame number, x, y, z are coordinates of the spatial position of the point cloud, and AM_difference is the average value of the difference value between the previous frame and the next frame in the angular FFT data of T frames. (Appendix 12) 12. An electronic device comprising: a memory having a computer program stored therein; and a processor, the processor being configured to execute the computer program so as to implement a method according to any one of claims 1 to 11. (Appendix 13) A storage medium having a computer readable program stored thereon, the computer readable program causing a computer to execute the method according to any one of claims 1 to 11 in an electronic device.

Claims

1. A device for detecting key points on a human body, comprising: an acquisition unit that detects an object using a radar to acquire a radar signal, and acquires a radar point cloud and zero Doppler FFT data based on the radar signal; A generating unit that generates a virtual point cloud based on the zero Doppler FFT data; a merging unit that performs point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud; a detector that performs keypoint detection on the merged point cloud using a keypoint detection model to obtain keypoints.

2. When generating a virtual point cloud based on the zero Doppler FFT data, the generation unit performing a two-dimensional Fourier transform on the zero Doppler FFT data to obtain angle FFT data; Calculate the average value of the difference between the previous frame and the next frame based on the angle FFT data of T frames, where T is greater than 0; A certain false alarm probability detection is performed in three directions, namely, the vertical angle, the horizontal angle, and the distance of the average value, and a position whose position value is all greater than a predetermined certain false alarm probability detection threshold is determined as a target position of the virtual point cloud; The apparatus of claim 1 , further comprising: converting the target positions into coordinates to obtain the virtual point cloud.

3. The apparatus of claim 1 , further comprising a filtering unit that filters the virtual point cloud based on a distance between the virtual point cloud and the radar point cloud to filter out virtual point clouds outside a predetermined distance range.

4. The filtering unit filters out a virtual point cloud whose distance from the center of the radar point cloud is equal to or greater than a first predetermined threshold, or The apparatus of claim 3 , wherein the filtering unit filters out virtual point clouds whose chamfer distance from the radar point cloud is equal to or greater than a second predetermined threshold.

5. A sorting unit that sorts the virtual point cloud according to a predetermined rule; The apparatus of claim 1 , further comprising: a selector for selecting a predetermined number of the top sorted virtual point clouds as a final result.

6. The predetermined rule includes an average value of difference values ​​between a previous frame and a next frame of angle FFT data of T frames, where T is greater than 0; the sorting unit sorts the virtual point cloud according to an average value of difference values ​​between a frame immediately before and a frame immediately after the T frames of angular FFT data; The apparatus of claim 5 , wherein the selector selects a top M virtual point clouds, where M is greater than 0.

7. the predetermined rule includes a distance between the virtual point cloud and a center of the radar point cloud; the sorting unit sorts the virtual point cloud according to a distance between the virtual point cloud and a center of the radar point cloud; The apparatus of claim 5 , wherein the selection unit selects M virtual point clouds with the smallest distances, where M is greater than 0.

8. When performing point cloud merging on the radar point cloud and the virtual point cloud, the merging unit The sum of the number of the radar point clouds and the number of the virtual point clouds is set as the number of the merged point clouds, Expanding the radar point cloud features and the virtual point cloud features to seven features corresponding to v, p, x, y, z, fID, and AM_difference, respectively; For the radar point cloud features, the values ​​of v, p, x, y, z, and fID are true values, and the value of AM_difference is 0 or the true value of the corresponding virtual point cloud; For the virtual point group features, the values ​​of v and p are 0, the values ​​of x, y, and z are true values, the value of fID is a fixed value, and the value of AM_difference is 0 or a true value; 2. The apparatus of claim 1, wherein v is Doppler, p is signal strength, fID is frame number, x, y, z are coordinates of a spatial position of a point cloud, and AM_difference is an average value of difference values ​​between a previous frame and a subsequent frame in angular FFT data of T frames.

9. A method for detecting key points on a human body, comprising: using a radar to sense an object and obtain a radar signal; and obtaining a radar point cloud and zero Doppler FFT data based on the radar signal; generating a virtual point cloud based on the zero Doppler FFT data; performing point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud; and performing keypoint detection on the merged point cloud using a keypoint detection model to obtain keypoints.

10. 10. An electronic device comprising: a memory having a computer program stored therein; and a processor, the processor being configured to execute the computer program so as to implement the method of claim 9.

Images