Key Point Correction Device and Method Based on Wireless Radar Signals

Abstract

An embodiment of the present application provides a key point correction device and method based on wireless radar signals. The method includes: perceiving an object through a radar to obtain point cloud data within a first time period; calibrating the validity of the point cloud of the current frame according to the number or proportion of the point clouds; calibrating the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or invalid key point group; retaining or discarding the key point group of the second time period according to the number or proportion of the invalid key point groups in the key point group within the second time period; and using a key point correction model to correct the key point group information of the current frame. The accuracy of the key point information of the current frame can be improved; in addition, by detecting the key points of an object (such as a human body) according to the radar point cloud, the action category is not limited, the required computing resources are small, and the detection accuracy is high; it is easy to implement, simple to operate, has strong anti-noise ability and high privacy protection.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of radar detection. Background Art

[0002] In the process of detecting human movements, key points of the human body can be detected, such as the head, neck, arms, feet, waist, etc. Human key point detection has a wide range of application scenarios and is a key technology for applications such as smart home, health monitoring, and behavior understanding.

[0003] Currently, video-based human key point detection technology is widely used. However, video seriously violates privacy and cannot be applied to private occasions. In addition, video-based key point detection is greatly affected by the environment (such as occlusion, lighting, smoke, etc.) and cannot function in the absence of light and occlusion scenarios; and it is also greatly affected by clothing, posture, and perspective.

[0004] Radar detects objects (such as the human body) through wireless signals, does not expose privacy, does not depend on external conditions such as lighting, and can work normally in some occlusion scenarios. Therefore, radar-based key point detection can make up for the deficiencies of video technology.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of clearly and completely explaining the technical solutions of the present application and facilitating the understanding of those skilled in the art. It cannot be considered that the above technical solutions are well-known to those skilled in the art just because these solutions are described in the background art part of the present application. Summary of the Invention

[0006] However, the inventors have found that conventional radar-based key point detection can only detect a small number of specific actions, which greatly limits its application scenarios. In addition, the coverage range of wireless signals is limited, and in the case of weak or lost wireless signals, it is easy to cause misdetection of key points, further reducing the accuracy of posture or action recognition.

[0007] To address at least one of the above technical problems, the embodiments of the present application provide a key point correction device and method based on wireless radar signals. Key points of an object (such as the human body) are detected according to radar point clouds, without limiting the action categories, requiring less computing resources, and having a high detection accuracy.

[0008] According to one aspect of the embodiments of the present application, a key point correction device based on wireless radar signals is provided, including:

[0009] A radar sensing unit that senses an object through a radar to obtain point cloud data within a first time period, where the first time period includes one frame or multiple frames;

[0010] A point cloud calibration unit that calibrates the validity of the point cloud of the current frame according to the number or proportion of the point cloud;

[0011] A key point calibration unit that calibrates the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or an invalid key point group;

[0012] A key point judgment unit that retains or discards the key point group in the second time period according to the number or proportion of the invalid key point groups in the key point group in the second time period to obtain input information for a key point correction model based on a neural network; and

[0013] A key point correction unit that corrects the key point group information of the current frame using the key point correction model.

[0014] According to another aspect of the embodiments of the present application, there is provided a key point correction method based on a wireless radar signal, including:

[0015] Perceiving an object through a radar to obtain point cloud data in a first time period, where the first time period includes one frame or multiple frames;

[0016] Calibrating the validity of the point cloud of the current frame according to the number or proportion of the point cloud;

[0017] Calibrating the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or an invalid key point group;

[0018] Retaining or discarding the key point group in the second time period according to the number or proportion of the invalid key point groups in the key point group in the second time period to obtain input information for a non-linear key point correction model; and

[0019] Correcting the key point group information of the current frame using the key point correction model.

[0020] One of the beneficial effects of the embodiments of the present application is that: by using the correlation of key point information to correct (calibrate or verify) the key point information detected based on wireless signals, the accuracy of the key point information of the current frame can be improved; in addition, detecting the key points of an object (such as a human body) according to radar point clouds can be independent of the action category, require less computing resources, and have a high detection accuracy; it is easy to implement, simple to operate, has strong anti-noise ability, and high privacy protection.

[0021] Referring to the following description and the accompanying drawings, specific embodiments of the embodiments of the present application are disclosed in detail, indicating the ways in which the principles of the embodiments of the present application can be adopted. It should be understood that the embodiments of the present application are not limited in scope thereby. Within the spirit and terms of the appended claims, the embodiments of the present application include many changes, modifications, and equivalents. Description of the Drawings

[0022] The accompanying drawings included are used to provide a further understanding of the embodiments of the present application, which form a part of the specification, illustrate the implementation manners of the present application, and explain the principles of the present application together with the written description. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other implementation manners can be obtained based on these drawings without creative efforts. In the drawings:

[0023] Figure 1 is a schematic diagram of a key point correction method based on wireless radar signals according to an embodiment of the present application;

[0024] Figure 2 is an example diagram of human key points according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of point cloud merging according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of point cloud validity calibration according to an embodiment of the present application;

[0027] Figure 5 is another schematic diagram of point cloud validity calibration according to an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of key point group cascading according to an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of judging a key point group according to an embodiment of the present application;

[0030] Figure 8 is another schematic diagram of judging a key point group according to an embodiment of the present application;

[0031] Figure 9 is a schematic diagram of key point correction according to an embodiment of the present application;

[0032] Figure 10 is a schematic diagram of a key point correction device based on wireless radar signals according to an embodiment of the present application;

[0033] Figure 11 is a schematic diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0034] Referring to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations, and equivalents falling within the scope of the appended claims.

[0035] In the embodiments of the present application, terms such as "first", "second", etc. are used to distinguish different elements in terms of name, but do not represent the spatial arrangement or time sequence of these elements, and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the related listed terms. Terms such as "comprising", "including", "having", etc. mean the presence of the stated features, elements, components, or assemblies, but do not exclude the presence or addition of one or more other features, elements, components, or assemblies.

[0036] In the embodiments of the present application, the singular forms "a", "the", etc. include the plural forms and should be broadly understood as "a kind" or "a class" rather than being limited to the meaning of "one"; in addition, the term "the" should be understood to include both the singular form and the plural form unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least partially according to...", and the term "based on" should be understood as "at least partially based on...", unless the context clearly indicates otherwise.

[0037] Features described and / or illustrated for one embodiment can be used in the same or similar manner in one or more other embodiments, combined with the features in other embodiments, or replace the features in other embodiments. The term "including / comprising" as used herein means the presence of features, whole things, steps, or components, but does not exclude the presence or addition of one or more other features, whole things, steps, or components.

[0038] In the embodiments of the present application, the radar can be a millimeter-wave (mmWave) radar, but is not limited thereto. The radar emits electromagnetic waves through a transmitting antenna and receives corresponding reflected waves (which can be called radar echo information) after being reflected by different objects. By analyzing the radar echo information, information such as the position of the object relative to the radar and the radial motion speed can be effectively extracted, and this information can meet the requirements of many application scenarios.

[0039] In the embodiments of the present application, the object to be detected can be people of various age groups. For example, it can be the elderly, children, or both the elderly and / or caregivers, children and / or guardians. The present application is not limited to this. The object to be detected can also be an animal with vital signs, or a robot without vital signs, etc. Hereinafter, the human body will be taken as an example for illustration.

[0040] Embodiments of the first aspect

[0041] The embodiments of the present application provide a key point correction method based on wireless radar signals. Figure 1 is a schematic diagram of the key point correction method based on wireless radar signals in the embodiments of the present application, as Figure 1 shown, the method includes:

[0042] 101. Sense an object through a radar to obtain point cloud data within a first time period, where the first time period includes one frame or multiple frames;

[0043] 102. Calibrate the validity of the point cloud of the current frame according to the number or ratio of the point cloud;

[0044] 103. Calibrate the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or invalid key point group;

[0045] 104. According to the number or ratio of the invalid key point groups in the key point group within a second time period, retain or discard the key point group of the second time period to obtain the input information of the key point correction model based on the neural network; and

[0046] 105. Use the key point correction model to correct the key point group information of the current frame.

[0047] It should be noted that the above Figure 1 only schematically illustrates the embodiments of the present application, but the present application is not limited to this. For example, the execution order between various operations can be appropriately adjusted. In addition, some other operations can be added or some of the operations can be reduced. Those skilled in the art can make appropriate modifications according to the above content, not limited to the above Figure 1 description.

[0048] In some embodiments, the radar senses the external space through wireless signals. The point cloud data output by the radar includes the distance, speed, position information, etc. of the object in the space detected by the radar. The radar periodically emits wireless signals for detection, and the point cloud information is also output periodically. The point cloud information output by the radar for one detection can be referred to as one frame of point cloud data.

[0049] In the embodiments of the present application, taking N consecutive frames of radar point cloud data as input, through the preprocessing of the radar point cloud data and the calculation of the action detection model, the key point information of the human body is output to represent the human action. The N-frame radar point cloud data can be represented by ; the frame sequence number i is arranged in chronological order. The larger the frame sequence number, the later the corresponding point cloud data appears. Therefore, P N represents the latest radar point cloud data.

[0050] In some embodiments, a frame of point cloud data P of the radar consists of several points, P = {p j , 1≤j≤n}, where n is the number of points included in this frame of point cloud, and p j is the j-th point. A point in the point cloud data is represented by p, p = (s, v, p, x, y, z), where s is the frame sequence number, v is the Doppler velocity relative to the radar, p is the signal intensity of this point, and (x, y, z) are its spatial coordinate values.

[0051] Therefore, the original spatial feature data directly obtained from the output signal of the radar can be used as the spatial feature data of the reflected point cloud, the Doppler velocity v directly obtained from the output signal of the radar can be used as the Doppler velocity feature data of the reflected point cloud, and the signal intensity p directly obtained from the output signal of the radar can be used as the reflection energy feature data of the reflected point cloud.

[0052] In addition, the original spatial feature data, Doppler velocity, and signal intensity obtained from the output signal of the radar can also be processed, and the processed data can be used as the spatial feature data, Doppler velocity feature data, and reflection energy feature data of the reflected point cloud. For example, the original spatial coordinate values (x0, y0, z0) of the radar output signal are processed such as translation and rotation, and the transformed values (x1, y1, z1) of the spatial coordinates obtained after processing are used as the first spatial feature data of the reflected point cloud, and so on; the present application is not limited to this.

[0053] In some embodiments, the human key points correspond to the main joints or organs of the human body, such as the nose, shoulders, elbows, wrists, hips, etc. The present application has no limitation on the selection of human key points. According to the requirements of specific applications, different human key points can be selected.

[0054] Figure 2 is an example diagram of the human key points in the embodiments of the present application. As Figure 2 shown, the human key point information can refer to the relative position relationship of the human joints or organs corresponding to the key points, and is represented by H = {(x k , y k , z k ), 1≤k≤m}, (x k , y k , zk ) is the spatial position information of the k-th key point. When there is only 2D position information, one dimension of the key point coordinates can be set to 0; for example, z can be set to 0, so that only (x, y) contains valid position information.

[0055] The above is a schematic description of the radar point cloud data, but the present application is not limited thereto.

[0056] In some embodiments, validity calibration can be performed for one frame. For example, the point cloud data within one frame is obtained. When the number of reflection points is less than the preset number threshold, the current frame is calibrated as invalid point cloud; otherwise, the current frame is calibrated as valid point cloud.

[0057] For example, the first time period can be set as one frame time. For a certain frame (the current frame), if the number of reflection points in this frame is 20, which is less than the preset number threshold (for example, 30), then this frame is calibrated as invalid point cloud; if the number of reflection points in this frame is 40, which is greater than or equal to the preset number threshold (for example, 30), then this frame is calibrated as valid point cloud.

[0058] In some embodiments, validity calibration can be performed for the merged multiple frames. For example, the point cloud data after merging multiple frames is obtained. When the total number of reflection points after merging multiple frames is less than the preset number threshold, or when the ratio of the number of reflection points in the current frame to the total number of reflection points is less than the preset ratio threshold, the current frame is calibrated as invalid point cloud; otherwise, the current frame is calibrated as valid point cloud.

[0059] Figure 3 is a schematic diagram of point cloud merging in an embodiment of the present application. For example, the first time period can be set as 10-frame time, that is, the point cloud data of 10 frames is merged. As Figure 3 shown, P1 to P 12 frames are 12 frames arranged in chronological order, where the signals of P3, P4, and P 12 are weak, and the signal of P8 is lost.

[0060] As Figure 3 shown, for the 12 frames from P1 to P 12 , P1 to P 10 can be merged in chronological order to form MP1, P2 to P 11 can be merged to form MP2, and P3 to P 12 can be merged to form MP3. Similarly, the merged point cloud information MP i (i = 1, ……, N) can be formed.

[0061] Figure 4 is a schematic diagram of point cloud validity calibration in an embodiment of the present application. As Figure 4As shown, for the current frame, the merged point cloud information is, for example, MP1. It can be determined whether PN is less than Th1 (as shown in 401), where PN is the total number of reflected points of this MP1, and Th1 is a preset number threshold. If PN is less than Th1, the current frame is calibrated as invalid point cloud (as shown in 402). If PN is greater than or equal to Th1, the current frame is calibrated as valid point cloud (as shown in 403).

[0062] Figure 5 is another schematic diagram of the point cloud validity calibration in the embodiments of the present application. As Figure 5 shown, for the current frame, the merged point cloud information is, for example, MP1. It can be determined whether Pr is less than Th2 (as shown in 501), where Pr = PNc / PN, PNc is the number of reflected points in the current frame, PN is the total number of reflected points of this MP1, and Th2 is a preset ratio threshold. If Pr is less than Th2, the current frame is calibrated as invalid point cloud (as shown in 502). If Pr is greater than or equal to Th2, the current frame is calibrated as valid point cloud (as shown in 503).

[0063] The above has given a schematic description of the calibration of the point cloud validity, but the present application is not limited thereto. By calibrating the validity of the point cloud information, the problem of false detection of key points caused by the inability to receive radar reflection signals for a long time or the weak radar signals received can be reduced or avoided, that is, the detection of invalid key points or the detection of key points with low precision can be reduced or avoided.

[0064] In some embodiments, a key point detection model based on a neural network is used to detect key points from the point cloud data after calibrating the validity, so as to obtain a set of key points of an object. For the specific content of key point detection, reference can also be made to related technologies.

[0065] In some embodiments, the validity of a set of key points can be calibrated according to the point cloud validity (for a certain frame). Specifically, the point cloud validity of the frame corresponding to a set of key points can be determined; and correspondingly (for example, in a one-to-one correspondence), it can be calibrated whether this set of key points is a set of valid key points or a set of invalid key points.

[0066] For example, for a frame with merged point cloud information of MP1 (abbreviated as MP1 frame), if the corresponding set of key points K1 is obtained according to the key point detection model, it can be considered that this MP1 frame corresponds to the set of key points K1. Correspondingly, if this MP1 frame is a valid point cloud, this set of key points K1 is a set of valid key points; if this MP1 frame is an invalid point cloud, this set of key points K1 is a set of invalid key points.

[0067] Thus, the validity calibration of the key point group can be performed. The above only schematically illustrates the validity calibration of the key point group, but the present application is not limited thereto. For example, the correspondence between the point cloud validity and the validity of the key point group may not be one-to-one, as long as the validity of the key point group can be determined based on the point cloud validity.

[0068] In some embodiments, according to the number or proportion of invalid key point groups in the key point group within the second time period, the key point group of the second time period is retained or discarded. Regarding the size of the second time period, it should not be too short or too long, and can be predetermined and adjusted according to the motion type, etc., to ensure the continuity of the target motion within a period of time.

[0069] In some embodiments, multiple calibrated key point groups within the second time period are obtained in chronological order. This process can be referred to as key point group cascading, or key point group binding, or key point group grouping, etc. Thus, the accuracy of key point detection in the current frame can be improved by leveraging the correlation between the preceding and succeeding actions.

[0070] Figure 6 is a schematic diagram of the key point group cascading in an embodiment of the present application. The merged point cloud information MP i Through the key point detection model, the corresponding key point group K can be obtained i (i = 1, …, N), and their validities correspond one-to-one.

[0071] After obtaining the key point group, for example, the second time period can be set to the time corresponding to 2 key point groups, that is, the second time period includes 2 key point groups. As Figure 6 shown, in chronological order, the key point groups K1 and K2 can be formed into MK1, and the key point groups K2 and K3 can be formed into MK2. Similarly, multiple key point groups MK within the second time period can be formed i (i = 1, …, N).

[0072] For example, K i (i = 1, …, N) represents the key point group information, including the key point group position information and the information indicating whether the key point group is a valid key point group or an invalid key point group, and MK i (i = 1, …, N) represents the key point group information within the second time period (which can be referred to as the cascaded key point group information), including multiple key point group position information and the information indicating whether each key point group is a valid key point group or an invalid key point group.

[0073] For another example, K i (i = 1, …, N) represents the key point group information, including the key point group position information and the corresponding point cloud validity of the key point group (such as the invalid point cloud ratio Pr), and MK i(i = 1, …, N) represents the key point group information (which can be called the cascaded key point group information) in the second time period, including the position information of multiple key point groups and the point cloud validity corresponding to each key point group (such as the invalid point cloud ratio Pr).

[0074] In some embodiments, according to the number or proportion of invalid key point groups in the key point group in the second time period, the key point group in the second time period is retained or discarded. Specifically, when the number of invalid key point groups in the second time period is less than the preset number threshold, or when the ratio of the number of invalid key point groups to the total number of key point groups is less than the preset proportion threshold, the key point group in the second time period is retained and used as the input information of the key point correction model; otherwise, the key point group in the second time period is discarded.

[0075] Figure 7 It is a schematic diagram for judging the key point group in the embodiment of the present application. As Figure 7 shown, for a key point group MK in a certain second time period i , the number Kn of invalid key point groups can be determined i (as shown in 701), and then it is judged whether Kn i is less than Th3 (as shown in 702), where Kn i is the number of invalid key point groups in the key point group MK in this second time period i , and Th3 is the preset number threshold; if Kn i is less than Th3, then the key point group MK in this second time period is retained i (as shown in 703), and used as the input information of the key point correction model (as shown in 704); if Kn i is greater than or equal to Th3, then the key point group MK in this second time period is discarded i (as shown in 705).

[0076] Figure 8 It is another schematic diagram for judging the key point group in the embodiment of the present application. As Figure 8 shown, for a key point group MK in a certain second time period i , the proportion Kr of invalid key point groups can be determined i (as shown in 801), and then it is judged whether Kr i is less than Th4 (as shown in 802), where Kr i = Kn i / Kn, Kn is the total number of key point groups in the key point group MK in this second time period i , Kn i is the number of invalid key point groups in the key point group MK in this second time period i , and Th4 is the preset proportion threshold;

[0077] As shown Figure 8 in, if Kr i is less than Th4, then the key point group MK i (as shown in 803) within the second time period is retained and used as the input information of the key point correction model (as shown in 804); if Kr i is greater than or equal to Th4, then the key point group MK i (as shown in 805) within the second time period is discarded.

[0078] The above is a schematic description of judging the key point group, but the present application is not limited thereto.

[0079] In the existing key point detection, if the number of detected invalid key points is large, it is difficult to make good use of the front and back action information, and instead, the accuracy of the current frame key point detection will be reduced due to a large number of invalid detections. In the embodiments of the present application, by judging the key point group within the second time period, the accuracy of the current frame key point detection can be corrected by making good use of the front and back actions, and the accuracy of the key point detection can be further improved.

[0080] The key point correction of the embodiments of the present application will be further described below.

[0081] In some embodiments, the key point correction model is used to obtain the correlation features between the key point group of the current frame and the key point groups before and / or after the current frame within the third time period; and the correlation features are used to correct the key point group information of the current frame to obtain the corrected key point group information.

[0082] In some embodiments, the key point correction model includes a non-linear function established based on the spatial position information of the key point groups before and / or after the current frame within the third time period, and the non-linear function is represented by the following formula:

[0083] FK i = f1(k) = f1([k i-t , …, k i , …, k i+t )

[0084] AK i = f2(FK i )

[0085] k i is the spatial position information of the key point group of the current frame, k i-t is the spatial position information of the key point group at time t before the current frame, k i+t is the spatial position information of the key point group at time t after the current frame, FK iThe association feature of the key point group of the currently acquired current frame with the key point groups before and / or after the current frame within the third time period, AK i Is the spatial position information of the key point group of the corrected current frame.

[0086] In some embodiments, the key point correction model is a fully connected neural network model, and the association feature includes a temporal feature and / or a spatial feature. That is, the temporal feature and / or spatial feature of the key points of the current frame can be extracted using a non-linear model by utilizing the position information of the key points of the previous and subsequent frames.

[0087] Figure 9 Is a schematic diagram of key point correction in an embodiment of the present application. For example, the third time period can be set to 2t + 1 frame times. As Figure 9 shown, assuming that the key point group of the current frame is K i , then the key point group K of the current frame can be utilized i , t key point groups K before the current frame i-t to K i-1 and t key point groups K after the current frame i+1 to K i+t , as the input MK of the key point correction model, and the key point correction model outputs AK ii Is the corrected key point information of the i-th group.

[0088] AK i = f(MK) = f(K i-m , …, K i , …, K i+l )

[0089] For example, a fully connected network including an input layer, an output layer, and one hidden layer is used to correct the key point information of an object:

[0090]

[0091] Among them, W 1 Is the weight information of the first layer of the neural network, W 2 Is the weight information of the second layer of the neural network, n is the number of neurons, and m is the number of key points included in the corrected key point group AK i of the i-th group.

[0092] The above has given a schematic description of obtaining the corrected key point group information through a fully connected network, but the present application is not limited thereto.

[0093] In some embodiments, the point cloud validity can also be utilized during key point correction. For example, according to the point cloud validity (the number PN of invalid point clouds or the proportion Pr of invalid point clouds) and the retained key point group, the key point group information is corrected.

[0094] Thus, according to the relevance of the forward and backward movement actions, using the key point group information (key point position information and / or the number or ratio of invalid point clouds) within a period of time as the input information of the non-linear model to correct the position information of the current frame key point group can further improve the accuracy of key point detection.

[0095] The above has schematically described the detection and correction methods and models. Embodiments of the present application can obtain one set / multiple sets of optimal parameters through a supervised training method; then apply the parameters to the detection model to perform operations on the input radar point cloud data to obtain the corresponding human key point information. Embodiments of the present application do not limit the specific training of the model. For example, SGD (Stochastic Gradient Descent) optimization, Adam (Adaptive Moment Estimation) optimization, etc. can be used.

[0096] The above only describes the steps or processes related to the present application, but the present application is not limited thereto. The action detection method may further include other steps or processes. For the specific content of these steps or processes, reference can be made to the prior art. In addition, the above only exemplarily describes embodiments of the present application by taking some structures of the action detection model as examples, but the present application is not limited to these structures and can also make appropriate modifications to these structures. The implementation manners of these modifications should all be included within the scope of embodiments of the present application.

[0097] The above embodiments only exemplarily describe embodiments of the present application, but the present application is not limited thereto and can also make appropriate modifications based on the above embodiments. For example, the above embodiments can be used alone, or one or more of the above embodiments can be combined.

[0098] As can be seen from the above embodiments, using the relevance of key point information to correct the key point information detected based on wireless signals can improve the accuracy of the key point information of the current frame; in addition, detecting the key points of an object (such as a human body) based on radar point clouds can be independent of the action category, requires less computing resources, and has a high detection accuracy; it is easy to implement, simple to operate, has strong anti-noise ability, and high privacy protection.

[0099] Embodiments of the second aspect

[0100] Embodiments of the present application provide a key point correction device based on wireless radar signals. The same content as that in the embodiments of the first aspect will not be repeated.

[0101] Figure 10 is a schematic diagram of the key point correction device based on wireless radar signals according to embodiments of the present application, asFigure 10 As shown in Figure 10 , the key point correction device 1000 based on wireless radar signals includes:

[0102] A radar sensing unit 1001 that senses an object through a radar to obtain point cloud data within a first time period, where the first time period includes one or more frames;

[0103] A point cloud calibration unit 1002 that calibrates the validity of the point cloud of the current frame according to the number or ratio of the point cloud;

[0104] A key point calibration unit 1003 that calibrates the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or an invalid key point group;

[0105] A key point judgment unit 1004 that retains or discards the key point group in the second time period according to the number or ratio of the invalid key point groups in the key point group within the second time period to obtain input information for the key point correction model based on a neural network; and

[0106] A key point correction unit 1005 that uses the key point correction model to correct the key point group information of the current frame.

[0107] In some embodiments, the point cloud calibration unit 1002 is configured to: obtain point cloud data within one frame, and in the case where the number of reflection points is less than a preset number threshold, calibrate the current frame as an invalid point cloud; otherwise, calibrate the current frame as a valid point cloud.

[0108] In some embodiments, the point cloud calibration unit 1002 is configured to: obtain point cloud data after merging multiple frames, and in the case where the total number of reflection points after merging multiple frames is less than a preset number threshold, or in the case where the ratio of the number of reflection points in the current frame to the total number of reflection points is less than a preset ratio threshold, calibrate the current frame as an invalid point cloud; otherwise, calibrate the current frame as a valid point cloud.

[0109] In some embodiments, the key point calibration unit 1003 is configured to: perform key point detection on the point cloud data after calibrating the validity by using a key point detection model based on a neural network to obtain a calibrated key point group.

[0110] In some embodiments, when the point cloud of the current frame corresponding to the key point group is an invalid point cloud, the key point group is an invalid key point group; when the point cloud of the current frame corresponding to the key point group is a valid point cloud, the key point group is a valid key point group.

[0111] In some embodiments, the key point determination unit 1004 is configured to: obtain multiple calibrated key point groups within the second time period in chronological order; when the number of invalid key point groups within the second time period is less than a preset number threshold, or when the ratio of the number of invalid key point groups to the total number of key point groups is less than a preset ratio threshold, retain the key point groups within the second time period and use them as the input information for the key point correction model; otherwise, discard the key point groups within the second time period.

[0112] In some embodiments, the key point correction unit 1005 is configured to: use the key point correction model to obtain the correlation features between the key point group of the current frame and the key point groups before and / or after the current frame within the third time period; and use the correlation features to correct the key point group information of the current frame to obtain the corrected key point group information.

[0113] In some embodiments, the key point correction model includes a non-linear function established based on the spatial position information of the key point groups before and / or after the current frame within the third time period.

[0114] The non-linear function is represented by the following formula:

[0115] FK i = f1(k) = f1([k i-t , …, k i , …, k i+t )

[0116] AK i = f2(FK i )

[0117] k i is the spatial position information of the key point group of the current frame, k i-t is the spatial position information of the key point group at time t before the current frame, k i+t is the spatial position information of the key point group at time t after the current frame, FK i is the correlation feature between the key point group of the current frame obtained and the key point groups before and / or after the current frame within the third time period, AK i is the spatial position information of the corrected key point group of the current frame.

[0118] In some embodiments, the key point correction unit 1005 is further configured to: correct the key point group information according to the point cloud validity and the retained key point groups.

[0119] It is worth noting that the above only describes the components or modules related to the present application, but the present application is not limited thereto. The key point correction device 1000 based on wireless radar signals may also include other components or modules, and the specific contents of these components or modules may refer to the relevant technology.

[0120] To keep it simple, Figure 10 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0121] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0122] It can be seen from the above embodiments that by using the correlation of key point information to correct the key point information detected based on wireless signals, the accuracy of the key point information of the current frame can be improved; in addition, the key points of an object (such as a human body) can be detected based on the radar point cloud without limiting the action category, requiring less computing resources and having a high detection accuracy; it is easy to implement, simple to operate, has strong noise resistance and high privacy protection.

[0123] Embodiments of the third aspect

[0124] The embodiment of the present application provides an electronic device, including the key point correction device 1000 based on wireless radar signals as described in the embodiment of the second aspect, and the content thereof is incorporated herein. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smart phone, etc., but the embodiment of the present application is not limited thereto.

[0125] Figure 11 Schematic diagram of an electronic device according to an embodiment of the present application. Figure 11 As shown, the electronic device 1100 may include: a processor (e.g., a central processing unit CPU) 1110 and a memory 1120; the memory 1120 is coupled to the central processing unit 1110. The memory 1120 may store various data; in addition, it may store a program 1121 for information processing, and the program 1121 may be executed under the control of the processor 1110.

[0126] In some embodiments, the functions of the key point correction device 1000 based on wireless radar signals are implemented by being integrated into the processor 1110. Among them, the processor 1110 is configured to implement the key point correction method based on wireless radar signals as described in the embodiments of the first aspect.

[0127] In some embodiments, the key point correction device 1000 based on wireless radar signals is separately configured from the processor 1110. For example, the key point correction device 1000 based on wireless radar signals can be configured as a chip connected to the processor 1110, and the functions of the key point correction device 1000 based on wireless radar signals are implemented through the control of the processor 1110.

[0128] For example, the processor 1110 is configured to perform the following controls: perceiving an object through a radar to obtain point cloud data within a first time period, where the first time period includes one frame or multiple frames; calibrating the validity of the point cloud of the current frame according to the number or proportion of the point cloud; calibrating the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or invalid key point group; retaining or discarding the key point group of the second time period according to the number or proportion of the invalid key point group in the key point group within the second time period to obtain input information for the key point correction model based on a neural network; and using the key point correction model to correct the key point group information of the current frame.

[0129] In addition, as Figure 11 shown, the electronic device 1100 may further include: an input / output (I / O) device 1130, a display 1140, etc.; among them, the functions of the above components are similar to those in the prior art and will not be elaborated here. It should be noted that the electronic device 1100 does not necessarily have to include Figure 11 all the components shown in Figure 11 ; in addition, the electronic device 1100 may further include components not shown in

[0130] Embodiments of the present application further provide a computer-readable program, where when the program is executed in an electronic device, the program causes the computer to execute the key point correction method based on wireless radar signals as described in the embodiments of the first aspect in the electronic device.

[0131] Embodiments of the present application further provide a storage medium storing a computer-readable program, where the computer-readable program causes the computer to execute the key point correction method based on wireless radar signals as described in the embodiments of the first aspect in an electronic device.

[0132] The above-described devices and methods of the present application can be implemented by hardware, or by a combination of hardware and software. The present application relates to such a computer-readable program that, when executed by a logic component, can cause the logic component to implement the devices or constituent components described above, or cause the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0133] The method / device described in combination with the embodiments of the present application can be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figures and / or a combination of one or more of the functional block diagrams can correspond to each software module of the computer program flow, and can also correspond to each hardware module. These software modules can respectively correspond to the respective steps shown in the figures. These hardware modules can be implemented by, for example, using a field-programmable gate array (FPGA) to solidify these software modules.

[0134] The software module can be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium can be coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium can be a component of the processor. The processor and the storage medium can be located in an ASIC. The software module can be stored in the memory of the mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if the device (such as a mobile terminal) uses a larger-capacity MEGA-SIM card or a large-capacity flash device, then the software module can be stored in the MEGA-SIM card or the large-capacity flash device.

[0135] One or more of the functional block diagrams described in the figures and / or a combination of one or more of the functional block diagrams can be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in the present application. One or more of the functional block diagrams described in the figures and / or a combination of one or more of the functional block diagrams can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication combination with a DSP, or any other such configuration.

[0136] The above description has been made in conjunction with specific embodiments of the present application. However, those skilled in the art should understand that these descriptions are exemplary and do not limit the protection scope of the present application. Those skilled in the art can make various variations and modifications to the present application based on the principles of the present application, and these variations and modifications are also within the scope of the present application.

[0137] Regarding the embodiments including the above embodiments, the following supplementary notes are also disclosed:

[0138] Supplementary Note 1. A key point correction method based on wireless radar signals, comprising:

[0139] Perceiving an object through a radar to obtain point cloud data within a first time period, the first time period including one frame or multiple frames;

[0140] Calibrating the validity of the point cloud of the current frame according to the number or proportion of the point cloud;

[0141] Calibrating the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or invalid key point group;

[0142] According to the number or proportion of the invalid key point groups in the key point group within a second time period, retaining or discarding the key point group of the second time period to obtain input information for a key point correction model based on a neural network; and

[0143] Using the key point correction model to correct the key point group information of the current frame.

[0144] Supplementary Note 2. The method according to Supplementary Note 1, wherein the calibrating the validity of the point cloud of the current frame according to the number or proportion of the point cloud includes:

[0145] Obtaining the point cloud data within one frame, and in the case where the number of reflection points is less than a preset number threshold, calibrating the current frame as an invalid point cloud; otherwise, calibrating the current frame as a valid point cloud.

[0146] Supplementary Note 3. The method according to Supplementary Note 1, wherein the calibrating the validity of the point cloud of the current frame according to the number or proportion of the point cloud includes:

[0147] Obtaining the point cloud data after merging multiple frames, and in the case where the total number of reflection points after merging multiple frames is less than a preset number threshold, or in the case where the ratio of the number of reflection points of the current frame to the total number of reflection points is less than a preset ratio threshold, calibrating the current frame as an invalid point cloud; otherwise, calibrating the current frame as a valid point cloud.

[0148] Supplementary Note 4. The method according to any one of Supplementary Notes 1 to 3, wherein a key point detection model based on a neural network is used to detect key points from the point cloud data after calibrating the validity to obtain a calibrated key point group.

[0149] Supplementary Note 5. According to the method described in Supplementary Note 4, wherein when the current frame point cloud corresponding to the key point group is an invalid point cloud, the key point group is an invalid key point group;

[0150] When the current frame point cloud corresponding to the key point group is a valid point cloud, the key point group is a valid key point group.

[0151] Supplementary Note 6. According to the method described in any one of Supplementary Notes 1 to 5, wherein the retaining or discarding of the key point group in the second time period according to the number or proportion of invalid key point groups in the key point group in the second time period includes:

[0152] Obtaining a plurality of calibrated key point groups in the second time period in chronological order;

[0153] When the number of invalid key point groups in the second time period is less than a preset number threshold, or when the ratio of the number of invalid key point groups to the total number of key point groups is less than a preset ratio threshold, retaining the key point group in the second time period and using it as the input information of the key point correction model; otherwise, discarding the key point group in the second time period.

[0154] Supplementary Note 7. According to the method described in any one of Supplementary Notes 1 to 6, wherein the using of the key point correction model to correct the key point group information of the current frame includes:

[0155] Using the key point correction model to obtain the associated features of the key point group of the current frame and the key point groups before and / or after the current frame in the third time period; and

[0156] Using the associated features to correct the key point group information of the current frame to obtain the corrected key point group information.

[0157] Supplementary Note 8. According to the method described in Supplementary Note 7, wherein the key point correction model includes a non - linear function established based on the spatial position information of the key point groups before and / or after the current frame in the third time period,

[0158] The non - linear function is represented by the following formula:

[0159] FK i = f1(k)= f1([k i-t ,…,k i ,…,k i+t )

[0160] AK i = f2(FK i )

[0161] ki is the spatial position information of the key point group of the current frame, k i-t is the spatial position information of the key point group at time t before the current frame, k i+t is the spatial position information of the key point group at time t after the current frame, FK i is the association feature of the key point group of the obtained current frame with the key point groups before and / or after the current frame within the third time period, AK i is the spatial position information of the corrected key point group of the current frame.

[0162] Remark 9. The method according to any one of Remarks 1 to 8, wherein the correcting the key point group information of the current frame by using the key point correction model further comprises:

[0163] correcting the key point group information according to the point cloud validity and the retained key point group.

[0164] Remark 10. An electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to implement the key point correction method based on wireless radar signals according to any one of Remarks 1 to 9.

[0165] Remark 11. A storage medium storing a computer-readable program, wherein the computer-readable program causes a computer to execute the key point correction method based on wireless radar signals according to any one of Remarks 1 to 9 in an electronic device.

Claims

1. A key point correction device based on wireless radar signals, characterized in that, The device includes: A radar sensing unit that senses an object through radar to obtain point cloud data within a first time period, where the first time period includes one or more frames; wherein, key point detection is performed based on the point cloud data to obtain a set of key points of the object; A point cloud calibration unit that calibrates the validity of the point cloud of the current frame according to the number or ratio of the point cloud; A key point calibration unit that calibrates the validity of the set of key points according to the point cloud validity to obtain a calibrated set of valid key points and / or an invalid set of key points; A key point judgment unit that retains or discards the set of key points in the second time period according to the number or ratio of the invalid set of key points in the set of key points in the second time period to obtain input information for a key point correction model based on a neural network; and A key point correction unit that uses the key point correction model to correct the information of the set of key points of the current frame.

2. The device according to claim 1, wherein, The point cloud calibration unit is used for: Obtaining point cloud data within one frame, and if the number of reflection points is less than a preset number threshold, calibrating the current frame as an invalid point cloud; otherwise, calibrating the current frame as a valid point cloud.

3. The device according to claim 1, wherein, The point cloud calibration unit is used for: Obtaining point cloud data after merging multiple frames, and if the total number of reflection points after merging multiple frames is less than a preset number threshold, or if the ratio of the number of reflection points in the current frame to the total number of reflection points is less than a preset ratio threshold, calibrating the current frame as an invalid point cloud; otherwise, calibrating the current frame as a valid point cloud.

4. The device according to claim 1, wherein, The key point calibration unit is used for: Performing key point detection on the point cloud data after calibrating the validity by using a key point detection model based on a neural network to obtain a calibrated set of key points.

5. The device according to claim 4, wherein, When the point cloud of the current frame corresponding to the set of key points is an invalid point cloud, the set of key points is an invalid set of key points; When the point cloud of the current frame corresponding to the set of key points is a valid point cloud, the set of key points is a valid set of key points.

6. The device according to claim 1, wherein, The key point judgment unit is used for: Obtaining multiple calibrated sets of key points within the second time period in chronological order; If the number of invalid sets of key points within the second time period is less than a preset number threshold, or if the ratio of the number of invalid sets of key points to the total number of sets of key points is less than a preset ratio threshold, retaining the set of key points within the second time period and using it as input information for the key point correction model; otherwise, discarding the set of key points within the second time period.

7. The device according to claim 1, wherein The key point correction unit is used for: Using the key point correction model to obtain the correlation features between the set of key points of the current frame and the sets of key points before and / or after the current frame within a third time period; And Using the correlation features to correct the information of the set of key points of the current frame to obtain corrected information of the set of key points.

8. The device according to claim 7, wherein, The key point correction model includes a non-linear function established based on the spatial position information of the sets of key points before and / or after the current frame within the third time period, The non-linear function is represented by the following formula: FK i = f1(k) = f1([k i-t ,…,k i ,…,k i+t ) AK i = f2(FK i ) k i is the spatial position information of the key point group of the current frame, k i-t is the spatial position information of the key point group at time t before the current frame, k i+t is the spatial position information of the key point group at time t after the current frame, FK i is the associated feature of the key point group of the current frame obtained and the key point groups before and / or after the current frame within the third time period, AK i is the spatial position information of the key point group of the corrected current frame.

9. The device according to claim 1, wherein The key point correction unit is further used for: Correcting the information of the set of key points according to the point cloud validity and the retained set of key points.

10. A key point correction method based on wireless radar signals, characterized in that, The method includes: Perceiving an object through radar to obtain point cloud data within a first time period, the first time period including one frame or multiple frames; wherein, performing key point detection on the point cloud data to obtain a key point group of the object; Calibrating the point cloud validity of the current frame according to the number or proportion of the point clouds; Calibrating the validity of the key point group according to the point cloud validity to obtain a calibrated valid key point group and / or invalid key point group; According to the number or proportion of the invalid key point groups in the key point group within a second time period, retaining or discarding the key point group of the second time period to obtain input information of a non-linear key point correction model; and Using the key point correction model to correct the key point group information of the current frame.

Images