Apparatus and method for correcting key points based on wireless radar signals

Abstract

To provide an apparatus for and a method of correcting a key point based on a radio radar signal according to an embodiment of the present invention.SOLUTION: A method according to the present invention has steps of sensing an object by a radar to acquire point group data in a first period, determining point group effectiveness of a current frame based on the number or proportion of point groups, determining effectiveness of a key point group based on the point group effectiveness to acquire an effective key point group and / or invalid key point group after the determination, holding or abandoning a key point group in a second period based on the number or proportion of the invalid key point group among the key point groups in the second period, and correcting the key point group information of the current frame by using a key point correction model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to the technical field of radar detection. [Background technology]

[0002] In the process of detecting human body movements, key points on the human body can be detected, such as the head, neck, arms, legs, waist, etc. Human body key point detection has a wide range of application scenarios and is an important technology in applications such as smart homes, health monitoring, and behavior understanding.

[0003] Currently, video-based human body keypoint detection technology is widely used. However, video capture seriously invades privacy and cannot be applied to private situations. Furthermore, video-based keypoint detection is heavily affected by the environment (e.g., occlusion, lighting, smoke, etc.), making it ineffective in dark and occluded scenarios. It is also heavily affected by clothing, pose, and viewpoint.

[0004] Radar detects objects (e.g., human bodies) through radio signals, does not expose privacy, is independent of lighting conditions, and can operate normally in partially occluded scenarios. Therefore, radar-based keypoint detection can compensate for the shortcomings of video technology.

[0005] It should be noted that the above description of the technical background is intended to provide a clear and complete understanding of the technical solutions of the present invention, and is described for the understanding of persons skilled in the art, and these technical solutions are merely described as part of the background technology of the present invention and are not well known by persons skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0006] However, according to the discovery of the inventors of the present invention, conventional radar-based keypoint detection can only detect a small number of specific actions, which greatly limits its application scenarios. In addition, the coverage of wireless signals is limited, and when the wireless signal is weak or lost, it is easy to cause false detection of keypoints, which further reduces the accuracy of posture or action recognition.

[0007] In view of at least one of the above technical problems, an embodiment of the present invention provides a keypoint correction device and method based on a wireless radar signal, which detects keypoints of an object (e.g., a human body) based on a radar point cloud, thereby not restricting the motion class, reducing the required computational resources, and improving the detection accuracy rate. [Means for solving the problem]

[0008] In one aspect of an embodiment of the present invention, there is provided a keypoint correction device based on a wireless radar signal, the device including: a radar detection unit that detects an object with a radar and acquires point cloud data within a first period, the first period including one frame or a plurality of frames; a point cloud determination unit that determines the validity of the point cloud for a current frame based on the number or proportion of point clouds; a keypoint determination unit that determines the validity of keypoint groups based on the point cloud validity and acquires valid keypoint groups and / or invalid keypoint groups after the determination; a keypoint judgment unit that retains or discards keypoint groups for the second period based on the number or proportion of invalid keypoint groups among the keypoint groups within a second period, and acquires input information for a keypoint correction model based on a neural network; and a keypoint correction unit that corrects keypoint group information for the current frame using the keypoint correction model.

[0009] In another aspect of the embodiment of the present invention, there is provided a keypoint correction method based on a wireless radar signal, the method including: sensing an object with a radar to acquire point cloud data within a first period, where the first period includes one frame or a plurality of frames; determining point cloud validity of a current frame based on a number or a proportion of point clouds; determining validity of keypoint groups based on the point cloud validity, and obtaining valid keypoint groups and / or invalid keypoint groups after the determination; retaining or discarding keypoint groups of the second period based on the number or a proportion of invalid keypoint groups among the keypoint groups in a second period, and obtaining input information for a nonlinear keypoint correction model; and correcting keypoint group information of the current frame using the keypoint correction model.

[0010] Some of the advantageous effects of the embodiments of the present invention are as follows: Using the relevance of keypoint information to correct (correct or verify) the keypoint information detected based on the wireless signal can improve the accuracy of the keypoint information of the current frame; Furthermore, detecting keypoints of an object (e.g., a human body) based on the radar point cloud can eliminate limitations on the motion class, reduce the required computational resources, and improve the detection accuracy; It is easy to implement, simple to operate, and has high noise resistance and privacy protection.

[0011] Certain embodiments of the present invention are disclosed in detail, as set forth in the following description and in the drawings, and are indicative of ways in which the principles of the present invention may be employed. However, the embodiments of the present invention are not intended to be limiting in scope. The present invention encompasses various alterations, modifications, and equivalents within the spirit and content of the appended claims. [Brief explanation of the drawings]

[0012] The drawings included herein are for understanding the embodiments of the present invention, constitute a part of this specification, and are for illustrating the embodiments of the present invention, and together with the written description, explain the principles of the present invention. Note that the drawings described herein are merely for illustrating the embodiments of the present invention, and those skilled in the art can easily derive other drawings based on these drawings. [Figure 1] FIG. 1 is a schematic diagram of a keypoint correction method based on a wireless radar signal according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram of one of the key points of the human body according to an embodiment of the present invention. [Figure 3] FIG. 2 is a schematic diagram of point cloud merging according to an embodiment of the present invention. [Figure 4] FIG. 1 is a schematic diagram of determining point cloud validity according to an embodiment of the present invention. [Figure 5] FIG. 10 is another schematic diagram of determining point cloud validity according to an embodiment of the present invention. [Figure 6] FIG. 2 is a schematic diagram of a cascade of keypoint groups according to an embodiment of the present invention; [Figure 7] FIG. 1 is a schematic diagram of determining a key point group according to an embodiment of the present invention; [Figure 8] FIG. 10 is another schematic diagram of determining key point groups according to an embodiment of the present invention; [Figure 9] FIG. 1 is a schematic diagram of a modification of key points according to an embodiment of the present invention. [Figure 10] 1 is a schematic diagram of a key point correction device based on a wireless radar signal according to an embodiment of the present invention; [Figure 11] 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] These and other features of the present invention will become apparent from the drawings and the following description. The specification and drawings disclose specific embodiments of the present invention, i.e., some embodiments consistent with the principles of the present invention. It should be noted that the present invention is not limited to the described embodiments, but includes all modifications, variations, and equivalents within the scope of the appended claims.

[0014] In the embodiments of the present invention, the terms "first" and "second" are used to distinguish different elements by name and do not refer to the spatial arrangement or temporal order of these elements, and these elements are not limited to these terms. The term "and / or" includes any one or more of the listed terms and combinations thereof. The terms "including," "including," and "having" refer to the presence of stated features, elements, elements, or components, but do not exclude the presence or addition of one or more other features, elements, elements, or components.

[0015] In the embodiments of the present invention, the singular forms "a," "the," etc. include the plural and mean "a kind" or "a class," and are not limited to "one." Furthermore, the term "said" includes both the singular and the plural unless the context clearly dictates otherwise. Furthermore, the term "according to" means "at least partially depending on," and the term "based on" means "at least partially based on," unless the context clearly dictates otherwise.

[0016] Features described and / or shown with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, in combination with features in other embodiments, or in place of features in other embodiments. The terms "comprising" or "including" refer to the presence of stated features, elements, components or members, but do not exclude the presence or addition of one or more other features, elements, components or members.

[0017] In an embodiment of the present invention, the radar may be, but is not limited to, a millimeter wave (mm Wave) radar. The radar emits electromagnetic waves through a transmitting antenna, and receives corresponding reflected waves (which may be referred to as radar echo wave information) after being reflected by various objects. By analyzing the radar echo wave information, information such as the object's radar position and radial movement speed can be effectively extracted, which can meet the needs of many application scenarios.

[0018] In an embodiment of the present invention, the object to be detected may be people of various ages, such as elderly people, children, elderly people and / or nursing staff, or children and / or guardians. The present invention is not limited thereto, and the object to be detected may be an animal with vital signs, a robot without vital signs, or the like. The following description will be given taking the human body as an example.

[0019] Example 1 An embodiment of the present invention provides a keypoint correction method based on a wireless radar signal. Figure 1 is a schematic diagram of a keypoint correction method based on a wireless radar signal according to an embodiment of the present invention. As shown in Figure 1, the method includes the following steps:

[0020] Step 101: Detect an object with a radar to obtain point cloud data within a first period, where the first period includes one frame or multiple frames.

[0021] Step 102: Determine the point cloud validity of the current frame based on the number or proportion of the point cloud.

[0022] Step 103: Determine the validity of the keypoint groups based on the point cloud validity, and obtain the determined valid keypoint groups and / or invalid keypoint groups.

[0023] Step 104: Based on the number or proportion of invalid keypoint groups among the keypoint groups in the second period, retain or discard the keypoint groups in the second period to obtain input information for a neural network-based keypoint correction model.

[0024] Step 105: Modify the keypoint group information of the current frame using the keypoint modification model.

[0025] Note that the above-described FIG. 1 merely exemplifies an embodiment of the present invention, and the present invention is not limited thereto. For example, the execution order of each step may be adjusted as appropriate, other steps may be added, or some steps may be deleted. Those skilled in the art may make appropriate modifications based on the above content, and are not limited to the description of the above-described FIG. 1.

[0026] In some embodiments, the radar senses the external space using radio signals, and the point cloud data output by the radar includes distance, speed, and position information of objects in the space detected by the radar. The radar periodically emits radio signals to perform detection, and the point cloud information is also periodically output. The point cloud information output by the radar each time it performs detection may be referred to as one frame of point cloud data.

[0027] In an embodiment of the present invention, N frames of continuous radar point cloud data are input, and after preprocessing of the radar point cloud data and calculation of a motion detection model, key point information of a human body is output to represent human motion. The radar point cloud data of N frames is (outside 1) The frame number i is sorted in chronological order, and the larger the frame number, the later the corresponding point cloud data appears. N represents the latest radar point cloud data.

[0028] In some embodiments, the point cloud data P of one frame of radar consists of several points, and P={pj , 1≦j≦n}, where n is the number of points included in the point cloud of the frame, and p j is the j-th point. A point in the point cloud data is represented by p, where p = (s,v,p,x,y,z), where s is the frame number, v is the relative radar Doppler velocity, p is the signal strength of the point, and (x,y,z) are its spatial coordinates.

[0029] Therefore, the original spatial characteristic data obtained directly from the radar output signal may be used as the spatial characteristic data of the reflection point group, the Doppler velocity v obtained directly from the radar output signal may be used as the Doppler velocity characteristic data of the reflection point group, and the signal intensity p obtained directly from the radar output signal may be used as the reflection energy characteristic data of the reflection point group.

[0030] Alternatively, the original spatial feature data, Doppler velocity, and signal intensity obtained from the radar output signal may be processed, and the processed data may be used as the spatial feature data, Doppler velocity feature data, and reflected energy feature data of the reflection point cloud. For example, the original spatial coordinate values ​​(x0, y0, z0) of the radar output signal may be subjected to processing such as translation and rotation, and the transformed spatial coordinate values ​​(x1, y1, z1) after processing may be used as the first spatial feature data of the reflection point cloud, but the present invention is not limited to this.

[0031] In some embodiments, the human body key points correspond to major joints or organs of the human body, such as the nose, shoulders, elbows, wrists, and waists. The present invention does not limit the selection of human body key points, and various human body key points may be selected according to the needs of specific applications.

[0032] FIG. 2 is a schematic diagram of a human body keypoint according to an embodiment of the present invention. As shown in FIG. 2, the human body keypoint information may refer to the relative positional relationship of the human body joints or organs corresponding to the keypoints, and H={(x k ,y k ,z k ), 1≦k≦m}, and (x k ,y k ,z k) are the coordinates of the k-th keypoint. If there is only two-dimensional location information, you can set the dimension of the keypoint coordinate to 0. For example, you can set z to 0, in which case only (x,y) contains valid location information.

[0033] Although the above has been a brief description of radar point cloud data, the present invention is not limited to this.

[0034] In some embodiments, a validity determination may be performed on a frame, for example, by acquiring point cloud data in a frame, and determining the current frame as an invalid point cloud if the number of reflection points is less than a predetermined number threshold, or determining the current frame as a valid point cloud if not.

[0035] For example, the first period may be set to the time of one frame. For a specific frame (current frame), if the number of reflection points in the frame is 20 and is less than a predetermined number threshold (e.g., 30), the frame is determined to be an invalid point cloud. If the number of reflection points in the frame is 40 and is greater than or equal to a predetermined number threshold (e.g., 30), the frame is determined to be a valid point cloud.

[0036] In some embodiments, a validity determination may be performed on the multiple frames after merging. For example, point cloud data is obtained after merging multiple frames, and if the total number of reflection points after merging multiple frames is smaller than a predetermined number threshold, or if the ratio of the number of reflection points of the current frame to the total number of reflection points is smaller than a predetermined ratio threshold, the current frame is determined to be an invalid point cloud; otherwise, the current frame is determined to be a valid point cloud.

[0037] 3 is a schematic diagram of a point cloud merging method according to an embodiment of the present invention. For example, the first period may be set to a time of 10 frames, that is, 10 frames of point cloud data may be merged. As shown in FIG. 3, frames P1 to P 12 are 12 frames sorted in time order, among which P3, P4 and P 12 The signal is weak and the P8 signal is lost.

[0038] As shown in Figure 3, P1 to P 12 For the 12 frames, P1 to P 10 are merged to form MP1, and P2~P 11 are merged to form MP2, and P3~P 12 Similarly, the merged point cloud information MP i (i=1,...,n) may be formed.

[0039] 4 is a schematic diagram of determining point cloud validity according to an embodiment of the present invention. As shown in FIG. 4, for the current frame, the merged point cloud information is, for example, MP1, and (as shown in step 401) P N It may be determined whether PN is less than Th1, where PN is the total number of reflection points of the MP1, and Th1 is a predetermined number threshold. If PN is less than Th1, the current frame is determined as an invalid point cloud (as shown in step 402), and if PN is greater than or equal to Th1, the current frame is determined as a valid point cloud (as shown in step 403).

[0040] 5 is another schematic diagram of determining point cloud validity according to an embodiment of the present invention. As shown in FIG. 5, for a current frame, the merged point cloud information is, for example, MP1, and it may be determined whether Pr is smaller than Th2 (as shown in step 501). Here, Pr = PNc / PN, PNc is the number of reflection points in the current frame, PN is the total number of reflection points in MP1, and Th2 is a predetermined ratio threshold. If Pr is smaller than Th2, the current frame is determined as an invalid point cloud (as shown in step 502); if Pr is greater than or equal to Th2, the current frame is determined as a valid point cloud (as shown in step 503).

[0041] Although the above is a brief description of determining the validity of a point cloud, the present invention is not limited thereto. By determining the validity of point cloud information, it is possible to reduce or avoid the problem of false keypoint detection caused by not receiving radar reflection signals for a long time or when the received radar signal is weak. That is, it is possible to reduce or avoid the detection of invalid keypoints or low-accuracy keypoints.

[0042] In some embodiments, a neural network-based keypoint detection model is used to perform keypoint detection on the point cloud data after validation to obtain a keypoint group of the object. For specific details of keypoint detection, please refer to the related art.

[0043] In some embodiments, the validity of a keypoint group (for a particular frame) may be determined based on point cloud validity. Specifically, the point cloud validity of a frame corresponding to a particular keypoint group may be determined, and then, depending on the point cloud validity (e.g., in a one-to-one correspondence), it may be determined whether the keypoint group is a valid keypoint group or an invalid keypoint group.

[0044] For example, if the merged point cloud information is a frame of MP1 (abbreviated as MP1 frame), and the corresponding keypoint group K1 is obtained according to the keypoint detection model, the MP1 frame may be considered to correspond to the keypoint group K1. Similarly, if the MP1 frame is a valid point cloud, the keypoint group K1 is a valid keypoint group, and if the MP1 frame is an invalid point cloud, the keypoint group K1 is an invalid keypoint group.

[0045] This allows validity determination to be performed on the keypoint group. The above has simply outlined a method for determining the validity of a keypoint group, but the present invention is not limited to this. For example, point cloud validity and keypoint group validity do not need to correspond one-to-one; it is sufficient if the validity of a keypoint group can be determined based on point cloud validity.

[0046] In some embodiments, the keypoint groups of the second period are retained or discarded based on the number or proportion of invalid keypoint groups among the keypoint groups within the second period. The size of the second period should not be too short or too long, and may be predetermined and adjusted according to the type of movement, etc. This can ensure the continuity of the target movement within a certain period of time.

[0047] In some embodiments, multiple keypoint groups determined within the second period are obtained in time order, which may be referred to as keypoint group cascading, keypoint group binding, or keypoint group grouping, etc. In this way, the correlation between previous and next actions can improve the accuracy of keypoint detection for the current frame.

[0048] 6 is a schematic diagram of a cascade of keypoint groups according to an embodiment of the present invention. i For each, the keypoint detection model finds the corresponding keypoint group K i (i=1,...,n) can be obtained, and the validity of each corresponds one-to-one.

[0049] After the keypoint groups are obtained, for example, a second period may be set to a time corresponding to two keypoint groups, i.e., the second period may include two keypoint groups. As shown in Figure 6, keypoint groups K1 and K2 may be formed as MK1 according to time order, and keypoint groups K2 and K3 may be formed as MK2. Similarly, keypoint groups MK within multiple second periods may be formed as MK1.i (i=1,...,N) may be formed.

[0050] For example, K i (i=1,...,N) represents keypoint group information, including the position information of the keypoint group and information on whether the keypoint group is a valid keypoint group or an invalid keypoint group. MK i (i=1,...,N) represents the keypoint group information within the second period (which may also be referred to as keypoint group information after cascading), including the position information of multiple keypoint groups and information on whether each keypoint group is a valid keypoint group or an invalid keypoint group.

[0051] As another example, K i (i=1,...,N) represents keypoint group information, including the position information of the keypoint group and the point cloud validity (e.g., the proportion of invalid point clouds Pr) corresponding to the keypoint group. i (i=1,...,N) represents keypoint group information within the second period (which may also be referred to as keypoint group information after cascading), including position information of multiple keypoint groups and point cloud validity (e.g., the proportion of invalid point clouds Pr) corresponding to each keypoint group.

[0052] In some embodiments, the keypoint groups in the second time period are retained or discarded based on the number or proportion of invalid keypoint groups among the keypoint groups in the second time period. Specifically, if the number of invalid keypoint groups in the second time period is less than a predetermined number threshold, or if the ratio of the number of invalid keypoint groups to the total number of keypoint groups is less than a predetermined ratio threshold, the keypoint groups in the second time period are retained as input information for the keypoint correction model; otherwise, the keypoint groups in the second time period are discarded.

[0053] 7 is a schematic diagram of determining a keypoint group according to an embodiment of the present invention. As shown in FIG. 7, a keypoint group MK in a certain second period i , the number of invalid keypoint groups Kn i and then (as shown in step 702) i It may be determined whether Kn is smaller than Th3. i is the keypoint group MK in the second period i where Th3 is a predetermined number threshold. If Kni is less than Th3, then (as shown in step 703) the number of invalid keypoint groups MK in the second period is i is reserved as input to the keypoint correction model (as shown in step 704), and Kn i If is greater than or equal to Th3, the keypoint group MK in the second period (as shown in step 705) i may be discarded.

[0054] 8 is another schematic diagram of determining a keypoint group according to an embodiment of the present invention. As shown in FIG. 8, a keypoint group MK in a certain second period i For each keypoint, the proportion of invalid keypoint groups Kr i and then (as shown in step 802) i It may be determined whether Kr is smaller than Th4. i =Kn i / Kn, Kn is the key point group MK within the second period i is the total number of i is the keypoint group MK in the second period i is the number of invalid keypoint groups among them, and Th4 is a predetermined percentage threshold.

[0055] As shown in Figure 8, Kr i If is smaller than Th4, the keypoint group MK in the second period (as shown in step 803) iis reserved as input to the keypoint correction model (as shown in step 804), and Kr i If is greater than or equal to Th4, the keypoint group MK in the second period (as shown in step 805) i Discard.

[0056] Although the above is a brief description of determining keypoint groups, the present invention is not limited to this.

[0057] In conventional keypoint detection, if a large number of invalid keypoints are detected, it is difficult to effectively utilize information about previous and following actions. Conversely, the accuracy of keypoint detection for the current frame is reduced due to the large number of invalid detections. In contrast, in the embodiment of the present invention, by determining the keypoint group within the second period, the accuracy of keypoint detection for the current frame can be corrected using previous and following actions, thereby further improving the accuracy of keypoint detection.

[0058] The following describes the modifications of key points according to an embodiment of the present invention.

[0059] In some embodiments, a keypoint modification model is used to obtain related features between the keypoint group of the current frame and the keypoint group before and / or after the current frame in the third period, and the related features are used to modify the keypoint group information of the current frame to obtain modified keypoint group information.

[0060] In some embodiments, the keypoint correction model includes a nonlinear function constructed based on spatial location information of the keypoint groups before and / or after the current frame within the third time period, the nonlinear function being expressed by the formula Free kick i =f1(k)=f1([k i-t ,…,k i ,…,k i+t ]) AK i =f2(FK i ) It is expressed using ki is the spatial location information of the keypoint group in the current frame, and k i-t is the spatial location information of the keypoint group before the current frame by time t, and k i+t is the spatial location information of the keypoint group after time t from the current frame, and FK i is the associated feature between the key point group of the current frame acquired and the key point group before and / or after the current frame in the third period, and AK i is the spatial location information of the key point group in the current frame after modification.

[0061] In some embodiments, the keypoint correction model is a neural network model based on full connectivity, and the related features include temporal features and / or spatial features, i.e., using the position information of keypoints in previous and subsequent frames, a nonlinear model may be used to extract the temporal features and / or spatial features of keypoints in the current frame.

[0062] 9 is a schematic diagram of a keypoint correction according to an embodiment of the present invention. For example, the third period may be set to the time of 2t+1 frames. As shown in FIG. 9, assuming that the keypoint group of the current frame is Ki, the keypoint group K of the current frame is i , the previous t keypoint groups K of the current frame i-t ~K i-1 , and t keypoint groups K after the current frame i+1 ~K i+t may be used as input MK for the keypoint correction model. i is output as the keypoint information of the i-th group after correction.

[0063] AK i =f(MK)=f(K i-m ,…,K i ,…,K i+l ) For example, a fully connected network including an input layer, an output layer, and one hidden layer is used to refine the keypoint information of the object.

number

[0064] where W 1 is the weight information of the first layer of the neural network, and W 2 is the weight information of the second layer of the neural network, n is the number of neurons, and m is the i-th keypoint group AK after correction. i is the number of keypoints contained in

[0065] Although the above briefly describes obtaining the modified keypoint group information using a fully connected network, the present invention is not limited to this.

[0066] In some embodiments, keypoint modification may utilize point cloud validity, for example, modifying keypoint group information based on point cloud validity (number of invalid point clouds PN or percentage of invalid point clouds Pr) and reserved keypoint groups.

[0067] This allows the accuracy of keypoint detection to be further improved by using keypoint group information (keypoint position information and / or the number or proportion of invalid point groups) within a certain period as input information for a nonlinear model to correct the position information of the keypoint group in the current frame depending on the relevance of the previous and next movements.

[0068] The above is an exemplary description of the detection and correction method and model. In an embodiment of the present invention, one or more sets of optimal parameters may be obtained through supervised training, and the parameters may be applied to the detection model to perform operations on the input radar point cloud data and obtain corresponding human body keypoint information. The embodiment of the present invention is not limited to the specific training of the model, and may use, for example, SGD (Stochastic Gradient Descent) optimization, Adam (Adaptive Moment Estimation) optimization, etc.

[0069] The above merely describes steps or processes related to the present invention, and the present invention is not limited thereto. The motion detection method may further include other steps or processes, and reference may be made to the prior art for the specific content of these steps or processes. Furthermore, the above merely exemplifies embodiments of the present invention by taking some structures of the motion detection model as examples, and the present invention is not limited to these structures, and appropriate modifications may be made to these structures, and these modifications should be included within the scope of the embodiments of the present invention.

[0070] The above-described embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto. Appropriate modifications may be made based on the above-described various embodiments. For example, the above-described embodiments may be used alone, or one or more of the above-described embodiments may be used in combination.

[0071] According to this embodiment, the relevance of keypoint information is used to correct the keypoint information detected based on the radio signal, thereby improving the accuracy of the keypoint information of the current frame. Furthermore, the keypoints of an object (e.g., a human body) are detected based on the radar point cloud, which does not restrict the motion class, reduces the required computational resources, and improves the detection accuracy. This method is easy to implement and operate, and has high noise resistance and privacy protection.

[0072] <Example 2> The embodiment of the present invention provides a keypoint correction device based on a wireless radar signal, and the description of the same contents as in the first embodiment will be omitted.

[0073] 10 is a schematic diagram of a keypoint correction device based on a wireless radar signal according to an embodiment of the present invention. As shown in FIG. 12, the keypoint correction device 1000 based on a wireless radar signal includes the following parts:

[0074] The radar sensing unit 1001 senses an object with a radar and acquires point cloud data within a first period, which includes one frame or multiple frames.

[0075] The point cloud determination unit 1002 determines the point cloud validity of the current frame based on the number or proportion of the point clouds.

[0076] The keypoint determination unit 1003 determines the validity of the keypoint groups based on the point group validity, and obtains the determined valid keypoint groups and / or invalid keypoint groups.

[0077] The keypoint determination unit 1004 retains or discards the keypoint groups of the second period based on the number or proportion of invalid keypoint groups among the keypoint groups in the second period, and obtains input information for the neural network-based keypoint correction model.

[0078] The keypoint modification unit 1005 modifies the keypoint group information of the current frame using the keypoint modification model.

[0079] In some embodiments, the point cloud determination unit 1002 acquires point cloud data in a frame, and if the number of reflection points is less than a predetermined number threshold, determines the current frame as an invalid point cloud; otherwise, determines the current frame as a valid point cloud.

[0080] In some embodiments, the point cloud determination unit 1002 obtains point cloud data after merging multiple frames, and determines the current frame as an invalid point cloud if the total number of reflection points after merging multiple frames is smaller than a predetermined number threshold, or if the ratio of the number of reflection points of the current frame to the total number of reflection points is smaller than a predetermined ratio threshold; otherwise, determines the current frame as a valid point cloud.

[0081] In some embodiments, the keypoint determination unit 1003 uses a keypoint detection model based on a neural network to perform keypoint detection on the point cloud data after validity determination, and obtains a determined keypoint group.

[0082] In some embodiments, if the point cloud of the current frame corresponding to the keypoint group is an invalid point cloud, the keypoint group is an invalid keypoint group, and if the point cloud of the current frame corresponding to the keypoint group is a valid point cloud, the keypoint group is a valid keypoint group.

[0083] In some embodiments, the keypoint determination unit 1004 obtains a plurality of determined keypoint groups within the second period in chronological order, and if the number of invalid keypoint groups within the second period is less than a predetermined number threshold, or if the ratio of the number of invalid keypoint groups to the total number of keypoint groups is less than a predetermined ratio threshold, retains the keypoint groups within the second period as input information for the keypoint correction model; otherwise, discards the keypoint groups within the second period.

[0084] In some embodiments, the keypoint modification unit 1005 uses the keypoint modification model to obtain associated features between the keypoint group of the current frame and the keypoint groups before and / or after the current frame within a third period, and uses the associated features to modify the keypoint group information of the current frame to obtain modified keypoint group information.

[0085] In some embodiments, the keypoint correction model includes a nonlinear function constructed based on spatial location information of the keypoint groups before and / or after the current frame within the third time period, the nonlinear function being expressed by the formula Free kick i =f1(k)=f1([k i-t ,…,k i ,…,k i+t ]) AK i =f2(FK i ) It is expressed using k i is the spatial location information of the keypoint group in the current frame, and k i-t is the spatial location information of the keypoint group before the current frame by time t, and k i+t is the spatial location information of the keypoint group after time t from the current frame, and FK i is the related feature between the acquired key point group of the current frame and the key point group before and / or after the current frame within the third period, and AK i is the spatial location information of the key point group in the current frame after modification.

[0086] In some embodiments, the keypoint modifier 1005 modifies the keypoint group information based on the point cloud validity and the reserved keypoint groups.

[0087] It should be noted that the above merely describes each component or module related to the present invention, and the present invention is not limited thereto. The keypoint correction device 1000 based on wireless radar signals may include other components or modules, and the specific contents of these components or modules may refer to the related art.

[0088] For simplicity, Figure 10 merely illustrates exemplary connection relationships or signal directions between each component or module, and it will be apparent to those skilled in the art that various related technologies, such as bus connections, may be used. The various components or modules described above may be realized by hardware devices, such as a processor and a memory, and the embodiments of the present invention are not limited thereto.

[0089] The above-described embodiments are merely illustrative of the embodiments of the present invention, and the present invention is not limited thereto. Appropriate modifications may be made based on the above-described various embodiments. For example, the above-described embodiments may be used alone, or one or more of the above-described embodiments may be used in combination.

[0090] According to this embodiment, the relevance of keypoint information is used to correct the keypoint information detected based on the radio signal, thereby improving the accuracy of the keypoint information of the current frame. Furthermore, the keypoints of an object (e.g., a human body) are detected based on the radar point cloud, which does not restrict the motion class, reduces the required computational resources, and improves the detection accuracy. This method is easy to implement and operate, and has high noise resistance and privacy protection.

[0091] Example 3 An embodiment of the present invention provides an electronic device including the keypoint correction device 1000 based on a wireless radar signal according to embodiment 2, the contents of which are incorporated herein by reference. The electronic device may be, for example, a computer, a server, a workstation, a laptop computer, a smartphone, etc., but the embodiment of the present invention is not limited thereto.

[0092] Fig. 11 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in Fig. 11, the electronic device 1100 according to the embodiment of the present invention includes a processor (e.g., a central processing unit (CPU)) 1110 and a memory 1120. The memory 1120 is connected to the processor 1110. The memory 1120 may store various data and may further store an information processing program 1121. The program 1121 is executed under the control of the processor 1110.

[0093] In one embodiment, the function of the wireless radar signal-based keypoint correction device 1000 may be integrated into the processor 1110. Here, the processor 1110 may be configured to implement the wireless radar signal-based keypoint correction method described in Example 1.

[0094] In another aspect, the keypoint correction device 1000 based on a wireless radar signal may be respectively arranged with a processor 1110, for example, the keypoint correction device 1000 based on a wireless radar signal may be a chip connected to the processor 1110 and configured to realize the functions of the keypoint correction device 1000 based on a wireless radar signal under the control of the processor 1110.

[0095] For example, the processor 1110 may be configured to perform the following steps: sensing an object with a radar to acquire point cloud data within a first period, where the first period includes one frame or multiple frames; determining point cloud validity for the current frame based on the number or proportion of point clouds; determining validity of keypoint groups based on the point cloud validity and obtaining valid and / or invalid keypoint groups after the determination; retaining or discarding keypoint groups for the second period based on the number or proportion of invalid keypoint groups among the keypoint groups in the second period, obtaining input information for a neural network-based keypoint correction model; and correcting keypoint group information for the current frame using the keypoint correction model.

[0096] As shown in Fig. 11, the electronic device 1100 may further include an input / output (I / O) device 1130, a display 1140, and the like. The functions of these components are similar to those of the prior art, and therefore, a description thereof will be omitted here. The electronic device 1100 does not need to include all of the components shown in Fig. 11. The electronic device 1100 may also include components not shown in Fig. 11, and may refer to the prior art.

[0097] An embodiment of the present invention provides a computer-readable program that, when executed in an electronic device, causes a computer to execute the keypoint correction method based on a wireless radar signal described in embodiment 1 in the electronic device.

[0098] An embodiment of the present invention further provides a storage medium storing a computer-readable program for causing a computer to execute the keypoint correction method based on a wireless radar signal described in embodiment 1 in an electronic device.

[0099] The above-described apparatus and method of the present invention may be realized by hardware or a combination of hardware and software. The present invention relates to a computer-readable program that, when executed by a logic unit, causes the logic unit to realize the above-described apparatus or components, or to implement the above-described various methods or steps. The present invention also relates to a storage medium for storing the above-described program, such as a hard disk, magnetic disk, optical disk, DVD, flash memory, etc.

[0100] The methods / apparatuses described with reference to the embodiments of the present invention may be implemented in hardware, software modules executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in the drawings, or one or more combinations of the functional block diagrams, may correspond to software modules in a computer program flow or hardware modules. These software modules may correspond to steps shown in the drawings. These hardware modules may be implemented by implementing these software modules in hardware, for example, using a field programmable gate array (FPGA).

[0101] The software module may be located in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, mobile hard disk, CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor reads information from or writes information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal or in a memory card inserted into the mobile terminal. For example, if a device (e.g., a mobile terminal) uses a relatively large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0102] One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any suitable combination thereof to perform the functions described herein. One or more of the functional blocks and / or one or more combinations of functional blocks depicted in the figures may be implemented, for example, with a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with a DSP communication, or any other configuration.

[0103] Although the present invention has been described above with reference to specific embodiments, the above description is merely illustrative and does not limit the scope of protection of the present invention. Various modifications and changes may be made to the present invention without departing from the spirit and principles of the present invention, and these modifications and changes also fall within the scope of the present invention.

[0104] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above-mentioned examples. (Appendix 1) A keypoint correction method based on a wireless radar signal, comprising: sensing an object with a radar to acquire point cloud data within a first period of time, the first period of time including one frame or multiple frames; determining point cloud validity for the current frame based on the number or percentage of point clouds; determining the validity of a keypoint group based on the point cloud validity, and obtaining a determined valid keypoint group and / or invalid keypoint group; According to the number or proportion of invalid keypoint groups among the keypoint groups in the second period, retain or discard the keypoint groups in the second period to obtain input information for a neural network-based keypoint correction model; and modifying keypoint group information of the current frame using the keypoint modification model. (Appendix 2) The step of determining point cloud validity of the current frame based on the number or proportion of the point clouds includes: 2. The method of claim 1, comprising the steps of acquiring point cloud data in a frame, and determining the current frame as an invalid point cloud if the number of reflection points is less than a predetermined number threshold, and otherwise determining the current frame as a valid point cloud. (Appendix 3) The step of determining point cloud validity of the current frame based on the number or proportion of the point clouds includes: 2. The method of claim 1, comprising the steps of: acquiring point cloud data after merging multiple frames; and determining the current frame as an invalid point cloud if the total number of reflection points after merging multiple frames is smaller than a predetermined number threshold, or if the ratio of the number of reflection points in the current frame to the total number of reflection points is smaller than a predetermined ratio threshold; otherwise, determining the current frame as a valid point cloud. (Appendix 4) 4. A method according to any one of claims 1 to 3, wherein a neural network-based keypoint detection model is used to perform keypoint detection on the point cloud data after validity has been determined, and a determined keypoint group is obtained. (Appendix 5) If the point cloud of the current frame corresponding to the key point group is an invalid point cloud, the key point group is an invalid key point group; 5. The method of claim 4, wherein a keypoint group is a valid keypoint group if the point cloud of the current frame corresponding to the keypoint group is a valid point cloud. (Appendix 6) The step of retaining or discarding the keypoint groups of the second time period based on the number or proportion of invalid keypoint groups among the keypoint groups in the second time period includes: Obtaining a plurality of determined keypoint groups within the second period in time order; retaining the keypoint groups in the second time period as input to the keypoint correction model if the number of invalid keypoint groups in the second time period is less than a predetermined number threshold, or if the ratio of the number of invalid keypoint groups to the total number of keypoint groups is less than a predetermined ratio threshold; and discarding the keypoint groups in the second time period otherwise. (Appendix 7) The step of modifying key point group information of the current frame using the key point modification model includes: using the keypoint correction model to obtain associated features between a group of keypoints in a current frame and a group of keypoints before and / or after the current frame within a third time period; and modifying key point group information of the current frame using the related features to obtain modified key point group information. (Appendix 8) the keypoint correction model includes a nonlinear function constructed based on spatial position information of a group of keypoints before and / or after the current frame within the third time period; The nonlinear function is expressed by the formula Free kick i =f1(k)=f1([k i-t ,…,k i ,…,k i+t ]) AK i =f2(FK i ) is represented using k i is the spatial location information of the keypoint group in the current frame, and k i-tis the spatial location information of the keypoint group before the current frame by time t, and k i+t is the spatial location information of the keypoint group after time t from the current frame, and FK i is the related feature between the acquired key point group of the current frame and the key point group before and / or after the current frame within the third period, and AK i is the spatial position information of the keypoint group of the current frame after correction. (Appendix 9) The step of modifying key point group information of the current frame using the key point modification model includes: 9. The method of any of claims 1 to 8, further comprising the step of modifying the keypoint group information based on the point cloud validity and the reserved keypoint groups. (Appendix 10) 10. An electronic device comprising: a memory in which a computer program is stored; and a processor, wherein the processor executes the computer program to realize a keypoint correction method based on a wireless radar signal according to any one of Supplementary Notes 1 to 9. (Appendix 11) 10. A storage medium having a computer-readable program stored thereon, the computer-readable program causing a computer to execute the method for keypoint correction based on a wireless radar signal according to any one of Supplementary Notes 1 to 9 in an electronic device.

Claims

1. A keypoint correction device based on a wireless radar signal, comprising: a radar sensing unit that senses an object using a radar and acquires point cloud data within a first period, the first period including one frame or a plurality of frames; a point cloud determination unit that determines the point cloud validity of the current frame based on the number or proportion of point clouds; a keypoint determination unit that determines the validity of a keypoint group based on the point cloud validity and acquires a determined valid keypoint group and / or an invalid keypoint group; a keypoint determination unit that retains or discards keypoint groups of a second time period based on the number or proportion of invalid keypoint groups among the keypoint groups in the second time period, and obtains input information for a neural network-based keypoint correction model; a keypoint modification unit that modifies keypoint group information of the current frame using the keypoint modification model.

2. The device according to claim 1 , wherein the point cloud determination unit acquires point cloud data within a frame, and if the number of reflection points is smaller than a predetermined number threshold, determines the current frame as an invalid point cloud, and if not, determines the current frame as a valid point cloud.

3. 2. The device of claim 1, wherein the point cloud determination unit acquires point cloud data after merging multiple frames, and determines the current frame as an invalid point cloud if the total number of reflection points after merging multiple frames is smaller than a predetermined number threshold, or if the ratio of the number of reflection points of the current frame to the total number of reflection points is smaller than a predetermined ratio threshold, and otherwise determines the current frame as a valid point cloud.

4. The apparatus according to claim 1 , wherein the keypoint determining unit uses a keypoint detection model based on a neural network to perform keypoint detection on the point cloud data after validity determination, and obtains a determined keypoint group.

5. If the point cloud of the current frame corresponding to the key point group is an invalid point cloud, the key point group is an invalid key point group; The apparatus of claim 4 , wherein a keypoint group is a valid keypoint group if the point cloud of the current frame corresponding to the keypoint group is a valid point cloud.

6. The key point determination unit Obtaining a plurality of determined keypoint groups within the second period in chronological order; 2. The apparatus of claim 1, wherein if the number of invalid keypoint groups in the second time period is less than a predetermined number threshold, or if the ratio of the number of invalid keypoint groups to the total number of keypoint groups is less than a predetermined ratio threshold, the keypoint groups in the second time period are retained as input information for the keypoint correction model; otherwise, the keypoint groups in the second time period are discarded.

7. The key point correction unit using the keypoint correction model to obtain associated features between a group of keypoints in the current frame and a group of keypoints before and / or after the current frame within a third time period; The apparatus of claim 1 , further comprising: modifying keypoint group information of the current frame using the related features to obtain modified keypoint group information.

8. the keypoint correction model includes a nonlinear function constructed based on spatial position information of keypoint groups before and / or after the current frame within the third time period; The nonlinear function is expressed by the formula FK ｉ =f １ (k)=f １ ([k ｉ－ｔ ,…,k ｉ ,…,k ｉ＋ｔ ]) A ｉ =f ２ (F) ｉ ) is represented using k ｉ is the spatial position information of the key point group of the current frame, and k ｉ－ｔ is the spatial position information of the key point group before the current frame by a time t, and k ｉ＋ｔ is the spatial position information of the key point group after the time t from the current frame, and FK ｉ is the associated feature between the acquired key point group of the current frame and the key point group before and / or after the current frame within a third period, and AK ｉ The apparatus according to claim 7 , wherein: is spatial position information of the key point group of the current frame after modification.

9. The apparatus of claim 1 , wherein the keypoint modifier modifies the keypoint group information based on the point cloud validity and reserved keypoint groups.

10. A keypoint correction method based on a wireless radar signal, comprising: sensing an object with a radar to acquire point cloud data within a first period of time, the first period of time including one frame or multiple frames; determining point cloud validity for the current frame based on the number or percentage of point clouds; determining the validity of a keypoint group based on the point cloud validity, and obtaining a determined valid keypoint group and / or an invalid keypoint group; According to the number or proportion of invalid keypoint groups among the keypoint groups in the second period, retain or discard the keypoint groups in the second period to obtain input information for a nonlinear keypoint correction model; and modifying keypoint group information of the current frame using the keypoint modification model.

Images