Human body key point detection method and device
By generating virtual point clouds and merging them with radar point clouds, the problem that radar point clouds cannot fully describe the targets are solved, and the accuracy of key point detection is improved.
Patent Information
- Application Number
- CN202311191177.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-15
- Publication Date
- 2025-03-18
AI Technical Summary
The point cloud output by the radar cannot fully describe the target in some cases, especially at resting areas, resulting in a decrease in the detection accuracy of key points.
By using zero Doppler FFT data to generate a virtual point cloud and merge the radar point cloud with the virtual point cloud to form a more complete point cloud to improve the accuracy of key point detection.
By combining radar point cloud and virtual point cloud, more complete target information is provided, the accuracy of human key point detection is improved and detection errors are reduced.
Smart Images
Figure CN119644281A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of radar detection technology. Background Art
[0002] The human key point detection method based on radar point cloud has been widely used because it does not infringe on privacy, can work in dark or partially occluded scenes, and can obtain the real spatial position of the human key points.
[0003] Figure 1 This is a schematic diagram of the application of the human key point detection method based on radar point cloud, such as Figure 1 As shown, when a human body moves, the corresponding point cloud can be obtained from the radar, and the key point information of the human body can be detected through the point cloud. The key point information of the human body can be used for motion detection, behavior analysis, fall detection, etc.
[0004] In motion detection and behavior analysis, the relative positions and statistical information of key points can be analyzed to determine the motion and behavior. In fall detection, the height information of the key points of the human body can be used to determine whether the target has fallen.
[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology part of the present application. Summary of the invention
[0006] The inventors found that the point cloud output by the radar (called radar point cloud or basic point cloud or original point cloud) is usually caused by the movement of the target and reflects the information of the moving part of the target. There is no point cloud for the static part with zero speed. Therefore, in some cases, the point cloud cannot fully describe the target. In addition, the quality of the input radar point cloud directly affects the detection accuracy of key points.
[0007] In order to solve the above-mentioned technical problems, the embodiments of the present application provide a method and device for key point detection to improve the detection accuracy.
[0008] According to one aspect of an embodiment of the present application, a method for detecting key points of a human body is provided, the method comprising:
[0009] Use radar to sense objects to obtain radar signals, and obtain radar point clouds and Doppler FFT data based on the signals;
[0010] generating a virtual point cloud based on the zero-Doppler FFT data;
[0011] Merging the radar point cloud and the virtual point cloud to obtain a merged point cloud;
[0012] The key point detection model is used to perform key point detection on the merged point cloud to obtain key points.
[0013] According to another aspect of an embodiment of the present application, a human body key point detection device is provided, the device comprising:
[0014] An acquisition unit, which uses a radar to sense an object to obtain a radar signal, and obtains a radar point cloud and zero Doppler FFT data according to the radar signal;
[0015] A generating unit, which generates a virtual point cloud based on the zero-Doppler FFT data;
[0016] A merging unit, which performs point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud;
[0017] A detection unit performs key point detection on the merged point cloud using a key point detection model to obtain key points.
[0018] According to another aspect of the embodiments of the present application, there is provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the human body key point detection method as described in the previous embodiment.
[0019] One of the beneficial effects of the embodiments of the present application is that human body key point detection is performed based on the radar point cloud corresponding to the moving part and the virtual point cloud corresponding to the stationary part, and the sum of the number of the radar point clouds and the number of the virtual point clouds is used as the number of the merged point clouds, which can provide more complete target information and thus improve the detection accuracy.
[0020] With reference to the following description and accompanying drawings, the specific implementation of the embodiment of the present application is disclosed in detail, indicating the way in which the principle of the embodiment of the present application can be adopted. It should be understood that the implementation of the present application is not limited in scope by the following text. Within the scope of the spirit and clauses of the attached claims, the implementation of the present application includes many changes, modifications and equivalents. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other implementation methods can be obtained based on these drawings without creative work. In the drawings:
[0022] Figure 1It is a schematic diagram of the application of the human body key point detection method based on radar point cloud;
[0023] Figure 2 This is a schematic diagram of radar sending and receiving signals;
[0024] Figure 3 It is a schematic diagram of obtaining radar point cloud and zero Doppler FFT data based on radar signals;
[0025] Figure 4 It is a schematic diagram of zero Doppler FFT data;
[0026] Figure 5 This is a schematic diagram of radar point cloud and Kinect detection results;
[0027] Figure 6 This is a schematic diagram of the Kinect detection true value and the predicted value of the key point detection model;
[0028] Figure 7 is a schematic diagram of a human body key point detection method according to an embodiment of the present application;
[0029] Figure 8 is a flow chart of a method for detecting key points of a human body according to an embodiment of the present application;
[0030] Fig. 9 This is a schematic diagram of generating a virtual point cloud based on zero-Doppler FFT data;
[0031] Fig.10 This is a schematic diagram of an example of AM_difference(e_bin,a_bin,range_bin);
[0032] Fig.11 A schematic diagram showing an example of the features of a radar point cloud
[0033] Fig.12 is a schematic diagram of an example of features of a virtual point cloud;
[0034] Fig.13 Schematic diagram of an example of a merged point cloud;
[0035] Fig.14 A schematic diagram comparing the model prediction value and the Kinet true value obtained by performing key point detection according to the method of an embodiment of the present application;
[0036] Fig.15 is a schematic diagram of a human body key point detection device according to an embodiment of the present application;
[0037] Fig.16 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0038] With reference to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which indicate some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents that fall within the scope of the appended claims.
[0039] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or time sequence of these elements, etc. These elements should not be limited by these terms, and the term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprise", "include", "have", etc. refer to the existence of the stated features, elements, components or components, but do not exclude the existence or addition of one or more other features, elements, components or components.
[0040] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.
[0041] Features described and / or shown for one embodiment may be used in one or more other embodiments in the same or similar manner, combined with features in other embodiments, or replace features in other embodiments. The term "includes / comprising" when used herein refers to the presence of a feature, an entirety, a step, or a component, but does not exclude the presence or addition of one or more other features, entireties, steps, or components.
[0042] In the embodiment of the present application, the radar may be a millimeter wave (mmWave) radar or a FMCW (Frequency Modulated Continuous Wave) radar, but the present application is not limited thereto. Figure 2 This is a schematic diagram of radar sending and receiving signals. Figure 2 As shown in the figure, the radar transmits electromagnetic waves (transmitting signals) through the transmitting antenna, and receives corresponding reflected waves (receiving signals, which can be called radar echo information) after being reflected by different targets. By analyzing the radar echo information, it is possible to effectively extract information such as the position of the target from the radar, the radial motion speed, etc., which can meet the needs of many application scenarios.
[0043] In an embodiment of the present application, a radar signal can be obtained by sensing an object using a radar. The radar signal is information in multiple dimensions, such as distance, Doppler (speed), angle, signal strength, etc.; based on the radar signal, a radar point cloud and zero Doppler FFT (Fast Fourier Transform) data can be obtained.
[0044] Figure 3 This is a schematic diagram of radar point cloud and zero Doppler FFT data obtained based on radar signals. Figure 4 This is a schematic diagram of zero Doppler FFT data. Figure 3 and Figure 4 As shown, by performing AD (analog-to-digital) conversion on the radar signal and performing range FFT and Doppler FFT on the converted signal, Doppler FFT data can be obtained, and zero Doppler FFT data (0-doppler-range FFT) can be extracted from the Doppler FFT data, as shown in FIG. Figure 4 In addition, by performing CFAR (Constant False-Alarm Rate) on the signal after Doppler FFT, and then performing angle FFT on the obtained signal, the radar point cloud is obtained.
[0045] In the above embodiments, usually, the point cloud output by the radar is caused by the movement of the target and reflects the information of the moving part of the target. There is no point cloud for the stationary part with zero speed. Therefore, in some cases, the point cloud cannot fully describe the target.
[0046] Figure 5 This is a schematic diagram of radar point cloud and Kinect detection results. Figure 5 As shown in (a), for a target sitting and waving, the radar point cloud can only reflect the movement of its hand (i.e., the moving part); Figure 5 As shown in (b), for a target bending over to pick up something, the radar point cloud can only reflect the movements of its head and arms (i.e., moving parts); Figure 5 As shown in (c), for a walking target, the radar point cloud can only reflect the movements of its waist and legs (i.e., moving parts); Figure 5 As shown in (d), for a fallen target or a prone target, the radar point cloud can only reflect the movement of its head and upper body (i.e., the moving parts). It can be seen that for the key point detection model, the quality of the input radar point cloud directly affects the detection accuracy of the key points. When the radar point cloud cannot fully describe the target, key point detection sometimes makes mistakes.
[0047] Figure 6 This is a schematic diagram of the Kinect detection true value and the predicted value of the key point detection model. Figure 6As shown in (a), the predicted value 61 of the key point detection model and the true value 62 of Kinect detection are completely inconsistent; Figure 6 As shown in (b), compared with the Kinect detection true value 64, the predicted value 63 of the key point detection model describes the target. However, the position of the target described by the predicted value 63 is higher than the true value 64 and is not accurate.
[0048] The various implementation methods of the embodiments of the present application are described below in conjunction with the accompanying drawings. In the embodiments of the present application, the detection target can be people of various ages, for example, it can be an elderly person, it can also be a child, it can also be an elderly person and / or a caregiver, a child and / or a guardian. The present application is not limited to this, and the detection target can also be an animal with life characteristics, or a robot without life characteristics, etc. The following is described by taking the human body as the detection target as an example.
[0049] Embodiments of the first aspect
[0050] The present application provides a method for detecting key points of a human body. Figure 7 is a schematic diagram of a human body key point detection method according to an embodiment of the present application, Figure 8 is a flow chart of the human body key point detection method according to the embodiment of the present application. Figure 7 and Figure 8 As shown, the method includes:
[0051] 701: Sense the target using a radar to obtain a radar signal, and obtain a radar point cloud and zero-Doppler FFT data according to the radar signal;
[0052] 702: Generate a virtual point cloud based on the zero Doppler FFT data;
[0053] 703: Merge the radar point cloud and the virtual point cloud to obtain a merged point cloud;
[0054] 704: Perform key point detection on the merged point cloud using a key point detection model to obtain key points of the target.
[0055] It is worth noting that the above Figure 7 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Figure 7 Records of.
[0056] According to the above embodiment, a radar point cloud and zero-Doppler FFT data are obtained according to the radar signal, a virtual point cloud is generated based on the zero-Doppler FFT data, and the radar point cloud and the virtual point cloud are merged. Thus, human body key point detection is performed based on the radar point cloud corresponding to the moving part and the virtual point cloud corresponding to the static part, which can provide more complete target information and thus improve the detection accuracy.
[0057] In the embodiment of the present application, there is no limitation on the method for acquiring radar point cloud and zero Doppler FFT data.
[0058] For example, you can Figure 3 The process shown in the figure obtains the radar point cloud, that is, AD conversion is performed on the radar signal; range FFT is performed on the radar signal after AD conversion; Doppler FFT is performed on the radar signal after range FFT; CFAR (constant false alarm detection) is performed on the radar signal after Doppler FFT; angle FFT is performed on the radar signal after CFAR to obtain the radar point cloud. The specific implementation process can refer to the relevant technology, and the description is omitted here.
[0059] For example, you can use Figure 3 The process shown and Figure 4 The method shown obtains zero Doppler FFT data, that is, AD conversion is performed on the radar signal; range FFT is performed on the radar signal after AD conversion; Doppler FFT is performed on the radar signal after range FFT to obtain Doppler FFT data, and then zero Doppler FFT data is extracted from the Doppler FFT data. The specific implementation process can refer to the relevant technology, and the description is omitted here.
[0060] In some embodiments, the virtual point cloud is obtained from zero-Doppler FFT data, for example, by Fig. 9 The method shown in the figure obtains a virtual point cloud from zero Doppler FFT data. Fig. 9 As shown, the method includes:
[0061] 910: Perform a two-dimensional Fourier transform on the zero-Doppler FFT data (0-doppler-range-FFT (ant_cnt, range_bin)) to obtain angle FFT data (angle-FFT (e_bin, a_bin, range_bin));
[0062] 920: Calculate the mean of the difference values of the previous and next frames based on the angle FFT data of the T frame (AM_difference(e_bin,a_bin,range_bin));
[0063] 930: Perform CFAR (constant false alarm detection) in the vertical angle, horizontal angle and distance directions of the mean value, and determine the position where the value of the position (e, a, r) is greater than the set CFAR threshold as the target position of the virtual point cloud;
[0064] 940: Convert the target position (e, a, r) into coordinates (x, y, z) to obtain the virtual point cloud.
[0065] In the above embodiment, the zero-Doppler FFT data can be expressed as 0-doppler-range-FFT (ant_cnt, range_bin), wherein ant_cnt is the number of antenna pairs, and range_bin is the number of frequency points of the range Fourier transform. The angle FFT data is obtained by performing a two-dimensional Fourier transform on 0-doppler-range-FFT (ant_cnt, range_bin).
[0066] In the above embodiment, the angle FFT data can be expressed as angle-FFT (e_bin, a_bin, range_bin), where e_bin and a_bin are vertical angle frequency points and horizontal angle frequency points, respectively, and the meaning of range_bin is the same as above, that is, the frequency point number of the range Fourier transform. For the meaning of angle FFT, please refer to the relevant technology, and the description is omitted here.
[0067] In the above embodiment, the average of the difference values of the angle FFT data of the previous and next frames can be calculated based on the continuous angle FFT data of T frames, which is expressed as AM_difference(e_bin, a_bin, range_bin). AM_difference(e_bin, a_bin, range_bin) is the average of the amplitude difference of the same distance frequency point, horizontal angle frequency point, and vertical angle frequency point over a period of time. Wherein, T is a natural number greater than 0, and this application does not limit the value of T.
[0068] In the above embodiment, constant false alarm detection is performed in the vertical angle, horizontal angle and distance directions of AM_difference (e_bin, a_bin, range_bin), and when the value of the position (e, a, r) is greater than the set CFAR threshold, the position is judged as the target position of the virtual point cloud, and the target position (e, a, r) is converted into coordinates (x, y, z) to obtain the coordinates of the virtual point cloud.
[0069] Fig.10 This is a schematic diagram of an example of AM_difference(e_bin,a_bin,range_bin), such as Fig.10As shown, in AM_difference(e_bin,a_bin,range_bin), e_bin=32, a_bin=32, and range_bin=128. According to the method of the above embodiment, constant false alarm detection is performed on AM_difference(e_bin,a_bin,range_bin) from three directions respectively. j ,a k ,r i ) is greater than the preset CFAR threshold, then the position (e j ,a k ,r i ) is determined as the target position, and the position (e j ,a k ,r i ) is converted into coordinates (x, y, z) to obtain the coordinates (x, y, z) of the virtual point cloud, thereby obtaining the virtual point cloud.
[0070] exist Fig.10 In the example, only one target position is taken as an example, but the present application is not limited thereto. All positions whose values are greater than a preset CFAR threshold are determined as target positions, and their corresponding coordinates are the coordinates of the virtual point cloud.
[0071] In the above embodiment, there is no restriction on the value of the CFAR threshold, which can be set according to experience or needs. In addition, different CFAR thresholds can be set for different targets, such as different groups of people; or different CFAR thresholds can be set for different radars, and so on.
[0072] In the above embodiments, the constant false alarm detection method is not limited, for example, a cell average detection (CA-CFAR) method or a minimum selection constant false alarm detection (SO-CFAR) method may be used. The present application is not limited thereto.
[0073] For example, for the CA-CFAR method, N reference cells are taken on both sides of the unit to be detected, and the average value of a total of 2N reference cells is calculated as the estimated value of the background clutter power; for another example, for the CO-CFAR method, the average values X and Y of the reference cells are calculated respectively, and the smaller average value of X and Y is selected as the estimated value of the background clutter power of the unit to be detected D. For the principles of CA-CFAR and CO-CFAR, reference can be made to the relevant technology, and the description is omitted here.
[0074] In the above embodiment, the same method or different methods can be used for constant false alarm detection in different directions. Fig.10For example, in the distance direction, SO-CFAR is used for constant false alarm detection, in the horizontal angle and vertical angle directions, CA-CFAR is used for constant false alarm detection, etc. The following table shows the CFAR thresholds corresponding to the above examples.
[0075] CFAR detection method (reference unit, protection unit), threshold distance SO-CFAR (6，4)，1 Horizontal Angle CA-CFAR (6，4)，1.5 Vertical Angle CA-CFAR (6，4)，1.5
[0076] In the above embodiment, the zero Doppler FFT data can be the zero Doppler FFT data within the entire distance range, or the zero Doppler FFT data within a specified distance range. The reason is that data that is too far away is unlikely to belong to the detection target, that is, this part of the data is of little significance for the key point detection of the target.
[0077] In one example, the range_bin in the above zero-Doppler FFT data may be determined by the following method.
[0078] S1: According to the original radar point cloud information, the distance frequency point set of the original radar point cloud is obtained, which is expressed as: R0 = {r i , i<=N}, where N is the number of original radar point clouds;
[0079] S2: In r i The distance frequency range r is selected within the neighborhood to generate the virtual point cloud. i _virtual, represented by: r i _virtual=(r i -r limit ,r i +r limit ), where r limit =Nr, for example, Nr=10; Nr is the number of distance frequency points, and Nr=10 means the range of 10 distance frequency points before and after the distance frequency point occupied by the point cloud of interest, that is, the range of 20 distance frequency points;
[0080] S3: Calculate the distance frequency point set used to generate the virtual point cloud, that is, all r i The union of _virtual is used as range_bin.
[0081] The determination method in the above example is only an example, and range_bin can also be determined by other methods.
[0082] In the above embodiment, optionally, Fig. 9 As shown, the method may also include:
[0083] 950: Filter the virtual point cloud according to the distance between the virtual point cloud and the radar point cloud, and filter out virtual point clouds outside a predetermined distance range.
[0084] For example, the virtual point clouds whose distance from the center of the radar point cloud is greater than or equal to the first predetermined threshold d1 are filtered out, that is, the distance between the virtual point cloud and the center of the radar point cloud is made less than d1, for example, less than 1.2 meters.
[0085] For another example, the virtual point clouds whose chamfer distance from the radar point cloud is greater than or equal to the second predetermined threshold d2 are filtered out, that is, the chamfer distance between the virtual point cloud and the radar point cloud group is made smaller than d2, for example, smaller than 1 meter.
[0086] The present application does not restrict the values of d1 and d2, which can be set according to the type of detection target and / or the type of radar and / or experience, and can be the same or different. Thus, the virtual point cloud that is unlikely to belong to the detection target is filtered out, which can reduce the amount of calculation in the subsequent key point detection process.
[0087] In the above embodiment, optionally, Fig. 9 As shown, the method may also include:
[0088] 960: Sort the virtual point cloud according to a predetermined rule;
[0089] 970: Select a predetermined number of virtual point clouds with the highest order as the final result.
[0090] In some examples, the predetermined rules include: the average of the difference values of the front and back frames of the angle FFT data of the T frame, that is, the virtual point clouds are sorted according to the average of the difference values of the front and back frames of the angle FFT data of the T frame, for example, the first M virtual point clouds are selected as supplementary points of the radar point cloud.
[0091] In other examples, the predetermined rules include: the distance between the virtual point cloud and the center of the radar point cloud, that is, sorting the virtual point clouds according to the distance between the virtual point cloud and the center of the radar point cloud, for example, selecting M virtual point clouds with the smallest distance.
[0092] In the above example, M is a natural number greater than 0, and this application does not limit the value of M.
[0093] It is worth noting that the above Fig. 9 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Fig. 9 Records of.
[0094] In the above embodiments, the radar point cloud and the virtual point cloud are merged, that is, the features and quantities are merged. In some examples, the sum of the number of radar point clouds and the number of virtual point clouds can be used as the number of merged point clouds, and the features of the radar point cloud and the features of the virtual point cloud are respectively expanded by the same bit to obtain the merged point cloud.
[0095] At present, the features of radar point cloud are six bits, namely x, y, z, v, p, fID, where x, y, z are the spatial position coordinates of radar point cloud, v is Doppler, p is signal strength, and fID is frame number. Fig.11 is a schematic diagram of an example of features of a radar point cloud.
[0096] In the above example, AM_difference (that is, the average of the differential values of the previous and next frames in the angle FFT data of the T frame) can be used as the seventh feature of the radar point cloud, where, among the seven features of the radar point cloud, the values of x, y, z, v, p, and fID are real values, that is, real values obtained according to the radar signal, and the value of AM_difference can be set to a fixed value, for example, set to 0, or set to the real value of the corresponding virtual point cloud.
[0097] In the above example, the virtual point cloud has the same seven features as the radar point cloud, and among the seven features of the virtual point cloud, the values of v and p can be set to fixed values, for example, set to 0; the values of x, y, and z can be real values, such as the coordinates of the virtual point cloud obtained by operation 902; the value of fID can be a fixed value, such as a constant C, C=0 or 1 or T+1, etc., where T can be the number of frames used to calculate AM_difference; the value of AM_difference can be a fixed value, for example, set to 0, or a real value, such as the mean of the difference values obtained by operation 920. Fig.12 is a schematic diagram of an example of features of a virtual point cloud.
[0098] According to the above embodiment, the point cloud is expanded from the basic point cloud [N, 6] to [N+M, 7], where N is the number of radar point clouds, M is the number of virtual point clouds, 6 and 7 are the number of feature bits, respectively, that is, the feature of the point cloud is expanded from 6 bits to 7 bits. Fig.13 is a schematic diagram of an example of the merged point cloud.
[0099] In an embodiment of the present application, a merged point cloud is obtained, that is, after the basic point cloud (radar point cloud obtained according to the radar signal) is expanded by the virtual point cloud, the key point detection model can be used to perform key point detection on the merged point cloud to obtain the key points of the detection target (such as the human body).
[0100] In the embodiments of the present application, there is no limitation on the structure and implementation method of the key point detection model, and specific details may be referred to related technologies.
[0101] The performance of the method of the embodiment of the present application is explained below by taking some action categories as examples.
[0102] In the following description, 9 action types are taken as an example, namely, action 21 walk to stand, action 22 stand to sit, action 23 walk to lie on bed, action 24 walk to lie on the floor, action 25 sit to lie on the floor, action 26 lie bed to lie on the floor, action 27 stand to lie on the floor, action 28 walk fall facing radar and action 29 walk fall far from radar.
[0103] The following table compares the mean absolute error (MAE) obtained by the method according to the embodiment of the present application with the MAR obtained by the existing method.
[0104]
[0105] It can be seen from the above table that according to an embodiment of the present application, assuming that based on 10 consecutive frames of zero-Doppler FFT data, 30 virtual point clouds are selected for merging through the amplitude difference of the angle FFT data, then: compared with the baseline, regardless of whether the seventh and sixth bits are 0 and 1 or 0 and 11 respectively, for each action category, the mean absolute error MAE is reduced, thereby improving the detection accuracy.
[0106] Fig.14 Schematic diagram of comparison between the model prediction value and Kinet true value obtained by performing key point detection according to the method of the embodiment of the present application. Fig.14 In the example shown in Figure 2, light-colored dots represent radar point clouds, and dark-colored triangles represent virtual point clouds.
[0107] like Fig.14 As shown in (a), for the falling action, compared with the model prediction value 141 (left side) obtained according to the existing method, the model prediction value 142 (right side) obtained according to the method of the embodiment of the present application is closer to the Kinet true value; Fig.14As shown in (b), for the standing action, compared with the model prediction value 143 (left side) obtained according to the existing method, the model prediction value 144 (right side) obtained according to the method of the embodiment of the present application is closer to the Kinet true value; Fig.14 As shown in (c), for the bending action, the model prediction value 146 (right side) obtained according to the method of the embodiment of the present application is closer to the Kinet true value than the model prediction value 145 (left side) obtained according to the existing method.
[0108] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.
[0109] According to the above embodiment, by using the virtual point cloud, human body key point detection is performed based on the radar point cloud corresponding to the moving part and the virtual point cloud corresponding to the static part, which can provide more complete target information, thereby improving the detection accuracy.
[0110] Embodiments of the second aspect
[0111] The embodiment of the present application provides a human body key point detection device, and its principle of solving the problem is similar to that of the embodiment of the first aspect, and the same contents will not be repeated here.
[0112] Fig.15 is a schematic diagram of a human body key point detection device according to an embodiment of the present application. Fig.15 As shown, the human body key point detection device 1500 of the embodiment of the present application includes:
[0113] An acquisition unit 1501 senses an object using a radar to obtain a radar signal, and obtains a radar point cloud and zero Doppler FFT data according to the radar signal;
[0114] A generating unit 1502, which generates a virtual point cloud based on the zero-Doppler FFT data;
[0115] A merging unit 1503 is configured to merge the radar point cloud and the virtual point cloud to obtain a merged point cloud; and
[0116] The detection unit 1504 performs key point detection on the merged point cloud using a key point detection model to obtain key points.
[0117] In some embodiments, the acquisition unit 1501 obtains the above zero Doppler FFT data by the following method.
[0118] S1: Perform AD conversion on the radar signal;
[0119] S2: Perform distance FFT on the radar signal after AD conversion;
[0120] S3: Performing Doppler FFT on the radar signal that has undergone range FFT to obtain the zero-Doppler FFT data.
[0121] In some embodiments, the acquisition unit 1501 obtains the above radar signal through the following method.
[0122] S1: Perform AD conversion on the radar signal;
[0123] S2: Perform distance FFT on the radar signal after AD conversion;
[0124] S3: Perform Doppler FFT on the radar signal after range FTT;
[0125] S4: Perform CFAR (constant false alarm detection) on the radar signal subjected to Doppler FFT;
[0126] S5: Performing angle FFT on the radar signal subjected to CFAR to obtain the radar point cloud.
[0127] In some embodiments, the generating unit 1502 generates a virtual point cloud based on the zero-Doppler FFT data, including:
[0128] S1: Perform a two-dimensional Fourier transform on the zero-Doppler FFT data (0-doppler-range-FFT(ant_cnt,range_bin)) to obtain angle FFT data (angle-FFT(e_bin,a_bin,range_bin));
[0129] S2: Based on the angle FFT data of the T frame, calculate the mean of the difference values of the previous and next frames (AM_difference(e_bin,a_bin,range_bin));
[0130] S3: Perform CFAR (constant false alarm detection) in the vertical angle, horizontal angle and distance directions of the above average value, and determine the position where the value of the position (e, a, r) is greater than the set CFAR threshold as the target position of the virtual point cloud;
[0131] S4: Convert the above target position into coordinates to obtain a virtual point cloud.
[0132] In some embodiments, Fig.15 As shown, the device 1500 also includes:
[0133] The filtering unit 1505 filters the virtual point cloud according to the distance between the virtual point cloud and the radar point cloud, and filters out the virtual point cloud outside a predetermined distance range.
[0134] In the above embodiment, the filtering unit 1505 can filter out the virtual point clouds whose distance from the center of the radar point cloud is greater than or equal to the first predetermined threshold d1; or, the filtering unit 1505 can also filter out the virtual point clouds whose chamfer distance from the radar point cloud is greater than or equal to the second predetermined threshold d2.
[0135] In some embodiments, Fig.15 As shown, the device 1500 also includes:
[0136] A sorting unit 1506, which sorts the virtual point cloud according to a predetermined rule;
[0137] A selection unit 1507 selects a predetermined number of virtual point clouds with the highest ranking as the final result.
[0138] In the above embodiment, the predetermined rules may include:
[0139] The average of the difference values of the front and back frames of the angle FFT data of the T frame, that is, the sorting unit 1506 sorts the virtual point clouds according to the average of the difference values of the front and back frames of the angle FFT data of the T frame, and the selection unit 1507 selects the first M virtual point clouds.
[0140] In the above embodiment, the predetermined rules may also include:
[0141] The distance between the center of the virtual point cloud and the radar point cloud, that is, the sorting unit 1506 sorts the virtual point clouds according to the distance between the center of the virtual point cloud and the radar point cloud, and the selection unit 1507 selects M virtual point clouds with the smallest distance.
[0142] In some embodiments, the merging unit 1503 performs point cloud merging on the radar point cloud and the virtual point cloud, including:
[0143] S1: The sum of the number of radar point clouds and the number of virtual point clouds is taken as the number of merged point clouds;
[0144] S2: Expand the features of the radar point cloud and the virtual point cloud to 7 bits respectively, namely: v, p, x, y, z, fID and AM_difference,
[0145] in,
[0146] In the features of the radar point cloud, the values of v, p, x, y, z, and fID are real values, and the value of AM_difference is 0 or the real value of the corresponding virtual point cloud;
[0147] In the features of the virtual point cloud, the values of v and p are 0, the values of x, y, and z are real values, the value of fID is a fixed value, and the value of AM_difference is 0 or the real value.
[0148] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The above device 1500 of the embodiment of the present application may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.
[0149] To keep it simple, Fig.15 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.
[0150] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.
[0151] It can be seen from the above embodiments that performing key point detection on the human body based on the radar point cloud corresponding to the moving parts and the virtual point cloud corresponding to the stationary parts can provide more complete target information, thereby improving the detection accuracy.
[0152] Embodiments of the third aspect
[0153] The embodiment of the present application provides an electronic device, including the apparatus 1500 as described in the embodiment of the second aspect, the contents of which are incorporated herein. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smart phone, etc., but the embodiment of the present application is not limited thereto.
[0154] Fig.16 Schematic diagram of an electronic device according to an embodiment of the present application. Fig.16 As shown, the electronic device 1600 may include: a processor (e.g., a central processing unit CPU) 1610 and a memory 1620; the memory 1620 is coupled to the central processing unit 1610. The memory 1620 may store various data; in addition, it may store a program 1621 for information processing, and execute the program 1621 under the control of the processor 1610.
[0155] In some embodiments, the functions of the apparatus 1500 described in the embodiment of the second aspect are integrated into the processor 1610 for implementation. The processor 1610 is configured to implement the method described in the embodiment of the first aspect.
[0156] In some embodiments, the device 1500 described in the embodiment of the second aspect is configured separately from the processor 1610. For example, the device 1500 described in the embodiment of the second aspect can be configured as a chip connected to the processor 1610, and the functions of the device 1500 described in the embodiment of the second aspect are implemented through the control of the processor 1610.
[0157] In addition, if Fig.16 As shown, the electronic device 1600 may also include: an input / output (I / O) device 1630 and a display 1640, etc.; wherein the functions of the above components are similar to those of the prior art and are not described in detail here. It is worth noting that the electronic device 1600 does not necessarily have to include Fig.16 In addition, the electronic device 1600 may also include Fig.16 For components not shown in the figure, reference may be made to the related art.
[0158] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program enables a computer to execute the method as described in the embodiment of the first aspect in the electronic device.
[0159] An embodiment of the present application further provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the method as described in the embodiment of the first aspect in an electronic device.
[0160] The above devices and methods of the present application can be implemented by hardware, or by hardware combined with software. The present application relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned devices or components, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.
[0161] The method / device described in conjunction with the embodiments of the present application may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to the various steps shown in the figure, respectively. These hardware modules may be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).
[0162] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if a device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.
[0163] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present application. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.
[0164] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.
[0165] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:
[0166] 1. A key point detection method, wherein the method comprises:
[0167] S1: Use radar to sense objects and obtain radar signals;
[0168] S2: Obtain radar point cloud and zero Doppler FFT data according to the radar signal;
[0169] S3: Generate a virtual point cloud based on the zero Doppler FFT data;
[0170] S4: merging the radar point cloud and the virtual point cloud to obtain a merged point cloud;
[0171] S5: Using a key point detection model to perform key point detection on the merged point cloud to obtain key points.
[0172] 2. The method according to Note 1, wherein, in S2, the zero Doppler FFT data is obtained by the following method:
[0173] Performing AD conversion on the radar signal;
[0174] Perform distance FFT on the radar signal after AD conversion;
[0175] Doppler FFT is performed on the radar signal that has undergone range FTT to obtain the zero-Doppler FFT data.
[0176] 3. The method according to Supplement 1, wherein, in S2, the radar point cloud is obtained by the following method:
[0177] Performing AD conversion on the radar signal;
[0178] Perform distance FFT on the radar signal after AD conversion;
[0179] Perform Doppler FFT on the radar signal that has undergone range FTT;
[0180] Perform CFAR (Constant False Alarm Rate) on the radar signal after Doppler FFT;
[0181] Angle FFT is performed on the radar signal subjected to CFAR to obtain the radar point cloud.
[0182] 4. The method according to Note 1, wherein S3 comprises:
[0183] S31: Perform a two-dimensional Fourier transform on the zero-Doppler FFT data (0-doppler-range-FFT (ant_cnt, range_bin)) to obtain angle FFT data (angle-FFT (e_bin, a_bin, range_bin));
[0184] S32: Based on the angle FFT data of frame T, calculate the average of the difference values of the previous and next frames (AM_difference(e_bin,a_bin,range_bin)), where T is greater than 0;
[0185] S33: performing CFAR (constant false alarm detection) in the vertical angle, horizontal angle and distance directions of the mean value, respectively, and determining the position where the value of the position (e, a, r) is greater than the set CFAR threshold as the target position of the virtual point cloud;
[0186] S34: Convert the target position into coordinates to obtain the virtual point cloud.
[0187] 5. The method according to Note 4, wherein:
[0188] The constant false alarm detection includes: unit average detection (CA-CFAR) or minimum selection constant false alarm detection (SO-CFAR).
[0189] 6. The method according to Note 4, wherein the method further comprises:
[0190] S35: Filtering the virtual point cloud according to the distance between the virtual point cloud and the radar point cloud, and filtering out virtual point clouds outside a predetermined distance range.
[0191] 7. The method according to Note 6, wherein:
[0192] Filter out virtual point clouds whose distance from the center of the radar point cloud is greater than or equal to a first predetermined threshold d1; or
[0193] The virtual point clouds whose chamfer distance from the radar point cloud is greater than or equal to a second predetermined threshold d2 are filtered out.
[0194] 8. The method according to Note 4, wherein the method further comprises:
[0195] S36: sorting the virtual point cloud according to a predetermined rule;
[0196] S37: Select a predetermined number of virtual point clouds with the highest order as the final result.
[0197] 9. The method according to Supplementary Note 8, wherein the predetermined rule comprises:
[0198] The average of the difference values of the previous and next frames of the angle FFT data of frame T,
[0199] The virtual point clouds are sorted according to the average of the difference values of the front and rear frames of the angle FFT data of the T frames, and the first M virtual point clouds are selected, where M is greater than 0.
[0200] 10. The method according to Supplementary Note 8, wherein the predetermined rule comprises:
[0201] The distance between the center of the virtual point cloud and the center of the radar point cloud,
[0202] The virtual point clouds are sorted according to the distance between the virtual point clouds and the centers of the radar point clouds, and M virtual point clouds with the smallest distances are selected, where M is greater than 0.
[0203] 11. The method according to Supplement 1, wherein S4 comprises:
[0204] The sum of the number of the radar point clouds and the number of the virtual point clouds is taken as the number of the merged point clouds;
[0205] The features of the radar point cloud and the features of the virtual point cloud are expanded to 7 bits respectively, namely: v, p, x, y, z, fID and AM_difference,
[0206] in,
[0207] In the features of the radar point cloud, the values of v, p, x, y, z, and fID are real values, and the value of AM_difference is 0 or the real value of the corresponding virtual point cloud;
[0208] In the features of the virtual point cloud, the values of v and p are 0, the values of x, y, and z are real values, the value of fID is a fixed value, and the value of AM_difference is 0 or a real value;
[0209] Where v is the Doppler, p is the signal strength, fID is the frame number, x, y, z are the spatial position coordinates of the point cloud, and AM_difference is the average of the difference values of the previous and next frames in the angle FFT data of the T frame.
[0210] 12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 11.
[0211] 13. A storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the method as described in any one of Notes 1 to 11 in an electronic device.
Claims
1. A human body key point detection device, wherein: The device comprises: An acquisition unit, which uses a radar to sense an object to obtain a radar signal, and obtains a radar point cloud and zero Doppler FFT data according to the radar signal; A generating unit, which generates a virtual point cloud based on the zero-Doppler FFT data; A merging unit, which performs point cloud merging on the radar point cloud and the virtual point cloud to obtain a merged point cloud; A detection unit performs key point detection on the merged point cloud using a key point detection model to obtain key points.
2. The device according to claim 1, wherein: The generating unit generates a virtual point cloud based on the zero Doppler FFT data, comprising: Performing a two-dimensional Fourier transform on the zero-Doppler FFT data to obtain angle FFT data; Based on the angle FFT data of frame T, calculate the average of the difference values of the previous and next frames, where T is greater than 0; In the vertical angle, horizontal angle and distance directions of the mean, constant false alarm detection is performed respectively, and the position whose value is greater than the set constant false alarm detection threshold is determined as the target position of the virtual point cloud; The target position is converted into coordinates to obtain the virtual point cloud.
3. The device according to claim 1, wherein: The device also includes: A filtering unit is configured to filter the virtual point cloud according to the distance between the virtual point cloud and the radar point cloud, and filter out the virtual point cloud outside a predetermined distance range.
4. The device according to claim 3, wherein: The filtering unit filters out virtual point clouds whose distance from the center of the radar point cloud is greater than or equal to a first predetermined threshold; or, The filtering unit filters out virtual point clouds whose chamfer distance from the radar point cloud is greater than or equal to a second predetermined threshold.
5. The device according to claim 1, wherein: The device also includes: A sorting unit, which sorts the virtual point cloud according to a predetermined rule; A selection unit selects a predetermined number of virtual point clouds that are ranked top as final results.
6. The device according to claim 5, wherein: The predetermined rules include: The mean of the difference values of the previous and next frames of the angle FFT data of frame T, where T is greater than 0; The sorting unit sorts the virtual point clouds according to the average of the difference values of the previous and next frames of the angle FFT data of the T frames, and the selection unit selects the first M virtual point clouds, where M is greater than 0.
7. The device according to claim 5, wherein: The predetermined rules include: The distance between the center of the virtual point cloud and the center of the radar point cloud, The sorting unit sorts the virtual point clouds according to the distance between the virtual point clouds and the centers of the radar point clouds, and the selection unit selects M virtual point clouds with the smallest distances, where M is greater than 0.
8. The device according to claim 1, wherein: The merging unit performs point cloud merging on the radar point cloud and the virtual point cloud, including: The sum of the number of the radar point clouds and the number of the virtual point clouds is taken as the number of the merged point clouds; The features of the radar point cloud and the features of the virtual point cloud are expanded to 7 bits respectively, namely: v, p, x, y, z, fID and AM_difference, in, In the features of the radar point cloud, the values of v, p, x, y, z, and fID are real values, and the value of AM_difference is 0 or the real value of the corresponding virtual point cloud; In the features of the virtual point cloud, the values of v and p are 0, the values of x, y, and z are real values, the value of fID is a fixed value, and the value of AM_difference is 0 or a real value; Where v is the Doppler, p is the signal strength, fID is the frame number, x, y, z are the spatial position coordinates of the point cloud, and AM_difference is the average of the difference values of the previous and next frames in the angle FFT data of the T frame.
9. A method for detecting key points of a human body, wherein: The method comprises: Using a radar to sense an object to obtain a radar signal, and obtaining a radar point cloud and zero-Doppler FFT data according to the radar signal; generating a virtual point cloud based on the zero-Doppler FFT data; Merging the radar point cloud and the virtual point cloud to obtain a merged point cloud; The key point detection model is used to perform key point detection on the merged point cloud to obtain key points.
10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: The processor is configured to execute the computer program to implement the method according to claim 9.