Key point detection method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and discloses a key point detection method and device, an electronic device and a storage medium. The method comprises the following steps: performing body part detection on each video frame in a video frame sequence to obtain position information of detection boxes in each video frame and categories corresponding to the detection boxes; determining a plurality of detection box sequences according to the position information of the detection boxes in different video frames in the video frame sequence and the categories corresponding to the detection boxes; connecting different detection box sequences according to the position information of a first detection box, the position information of a last detection box and time sequence information of video frames in which the detection boxes in the detection box sequences are located in the video frame sequence, to obtain at least one combined detection box sequence; and performing key point detection on pixel regions surrounded by the detection boxes in the combined detection box sequence to obtain a key point sequence corresponding to the combined detection box sequence. The application can guarantee the effectiveness of the obtained key point sequence for action tracking.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a key point detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Keypoint detection, as a prerequisite for tasks such as motion transfer and behavior detection, has attracted widespread attention. Related technologies utilize keypoint detection on each frame of a video frame sequence to determine and track human actions within that sequence. However, these technologies sometimes fail to accurately track human actions within the video frame sequence. Summary of the Invention

[0003] In view of the above problems, this application proposes a key point detection method, apparatus, electronic device and storage medium to solve the problem that key point sequences cannot effectively track the actions of the human body in a video frame sequence in related technologies.

[0004] According to one aspect of the embodiments of this application, a keypoint detection method is provided, comprising: performing body part detection on each video frame in a video frame sequence to obtain position information of detection boxes in each video frame and a category corresponding to each detection box, wherein the category corresponding to the detection box is used to indicate the body part to which the pixel area enclosed by the detection box belongs; determining multiple detection box sequences based on the position information of the detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box, wherein the pixel areas enclosed by different detection boxes in a detection box sequence belong to the same body part; connecting different detection box sequences based on the position information of the first detection box in each detection box sequence, the position information of the last detection box in each detection box sequence, and the temporal information of the video frame in which the detection box in the detection box sequence is located in the video frame sequence to obtain at least one combined detection box sequence; and performing keypoint detection on the pixel areas enclosed by each detection box in each combined detection box sequence for each combined detection box sequence to obtain a keypoint sequence corresponding to the combined detection box sequence.

[0005] According to one aspect of the embodiments of this application, a key point detection apparatus is provided, comprising: a detection module, configured to perform body part detection on each video frame in a video frame sequence, and obtain position information of detection boxes in each video frame and a category corresponding to each detection box, wherein the category corresponding to the detection box is used to indicate the body part to which the pixel area enclosed by the detection box belongs; a detection box sequence determination module, configured to determine multiple detection box sequences based on the position information of the detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box, wherein the pixel areas enclosed by different detection boxes in a detection box sequence belong to the same body part; a connection module, configured to connect different detection box sequences based on the position information of the first detection box in each detection box sequence, the position information of the last detection box, and the timing information of the video frame in the video frame sequence where the detection box is located, to obtain at least one combined detection box sequence; and a key point detection module, configured to perform key point detection on the pixel areas enclosed by each detection box in the combined detection box sequence for each combined detection box sequence, to obtain a key point sequence corresponding to the combined detection box sequence.

[0006] According to one aspect of the embodiments of this application, an electronic device is provided, including: a processor; a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, the key point detection method described above is implemented.

[0007] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the key point detection method as described above.

[0008] According to one aspect of the embodiments of this application, a computer program product is provided, which includes computer instructions that, when executed by a processor, implement the key point detection method described above.

[0009] In this application, the potential for missed or false detections during body part detection, which could lead to a detection box sequence that cannot effectively track the movement of body parts in a video frame sequence, is fully considered. Therefore, after obtaining the detection box sequence, the different detection box sequences are connected by combining the position information of the first and last detection boxes in each sequence, the position information of the last detection box, and the timing information of the video frame containing the detection box in the sequence within the video frame sequence. This ensures that the resulting combined detection box sequence can accurately track the movement of body parts in the video frame sequence, thereby ensuring that the resulting keypoint sequence accurately reflects the actions of each body part in the video frame sequence and guarantees the correspondence between the obtained keypoint sequence and the video frame sequence. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application.

[0012] Figure 2 This is a flowchart illustrating a key point detection method according to an embodiment of this application.

[0013] Figure 3 yes Figure 2 Step 220 in one embodiment is shown in a flowchart.

[0014] Figure 4 The diagram above illustrates a sequence of detection boxes.

[0015] Figure 5 yes Figure 2 Step 230 in one embodiment is shown in a flowchart.

[0016] Figure 6 An illustrative diagram of key points on a face is shown.

[0017] Figure 7 This is a flowchart illustrating the steps following step 240 according to an embodiment of this application.

[0018] Figure 8 This is a block diagram of a key point detection device according to an embodiment of this application.

[0019] Figure 9 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of this application is shown. Detailed Implementation

[0020] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0021] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0022] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0023] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0024] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of this application according to an embodiment of this application, such as... Figure 1 As shown, the application scenario includes a terminal 110 and an electronic device 120 that communicates with the terminal 110. The electronic device 120 can be a server (physical server or cloud server) or other device with certain computing capabilities.

[0026] Terminal 110 can be a camera or other device with image acquisition capabilities, without specific limitations. Thus, terminal 110 can capture video from a movable object, such as a person or animal, without further specific limitations. Afterwards, terminal 110 can send the captured video to electronic device 120, which performs keypoint detection based on the video according to the method of this application to obtain a keypoint video sequence. Specifically, electronic device 120 performs keypoint detection according to the following process: detecting body parts in the video frames; determining multiple detection box sequences; connecting different detection box sequences; and performing keypoint detection on the combined detection box sequences. The implementation details of each step in the above process are described below.

[0027] In other embodiments, terminal 110 may also be other devices that can display an interactive interface, so that the user can send a video to be identified for key points to electronic device 120 through the interactive interface of terminal 110. The video may be a video pre-stored in terminal 110.

[0028] The implementation details of the technical solutions in the embodiments of this application are described in detail below:

[0029] Figure 2 This is a flowchart illustrating a keypoint detection method according to an embodiment of this application, which can be executed by an electronic device with processing capabilities. (Refer to...) Figure 2 As shown, the method includes at least steps 210 to 240, which are described in detail below:

[0030] Step 210: Perform body part detection on each video frame in the video frame sequence to obtain the position information of the detection box in each video frame and the category corresponding to each detection box. The category corresponding to the detection box is used to indicate the body part to which the pixel area enclosed by the detection box belongs.

[0031] The video frame sequence can originate from real-time captured video or from pre-stored video. Specifically, body parts can be detected in each video frame of the video frame sequence using an object detection model. The object detection model can be a model constructed using one or more neural networks, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, etc., without specific limitations. In specific embodiments, the object detection model can be NanoDet (a single-stage anchor-free object detection model similar to FCOS (Fully Convolutional One-Stage Object Detection), Faster R-CNN (Region-Convolutional Neural Network), YOLO, etc.

[0032] In this application, the object detection model can detect one or more body parts. For example, body parts can be the head, body, hands (left hand, right hand), feet (left foot, right foot), etc. Of course, the granularity of body part segmentation is not limited to this; it can be segmented at a coarser or finer granularity. When there are multiple body parts to be detected, a separate object detection model can be trained for each body part (e.g., a model for detecting the head, a model for detecting the body, etc.), or a single object detection model can be used to detect multiple body parts.

[0033] To ensure the accuracy of body part detection by the object detection model, it needs to be trained beforehand using initial training data. This initial training data includes a first sample image and its corresponding label information. The label information indicates the position of the bounding boxes surrounding body parts (head, body, left hand, right hand, left foot, right foot, etc.) within the first sample image. During training, the first sample image is input into the object detection model, which performs body part detection and outputs the detection results. These results include the position information of the detection boxes surrounding the body parts and the corresponding category information. The category information indicates the body part to which the pixel region encompassed by the detection box belongs. Then, based on the position and category information of the detection boxes in the body part detection results, as well as the label information of the first sample image, the loss function is calculated. The parameters of the object detection model are then adjusted in reverse based on the loss value until the training termination condition is met. The training termination condition can be either the number of iterations of the object detection model reaching a threshold or the convergence of the loss function of the object detection model. The loss function of the object detection model can be a cross-entropy loss function, a mean squared error loss function, a smoothed mean absolute error loss function, etc., without specific limitations. After training, applying the object detection model to body part detection can ensure the accuracy of the location information of the detection boxes and the corresponding categories in the detected images (video frames).

[0034] Step 220: Based on the position information of the detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box, determine multiple detection box sequences, wherein the pixel regions surrounded by different detection boxes in a detection box sequence belong to the same body part.

[0035] Different detection boxes in the detection box sequence originate from different video frames, and the video frames containing two adjacent detection boxes in the detection box sequence are consecutive in the video frame sequence.

[0036] The objects (such as people) presented in the video frame sequence may be in an active state. Therefore, the position of the same body part of the object is different in different video frames. Therefore, in this application, the detection boxes corresponding to each body part in each video frame are associated to obtain a detection box sequence, so as to track the motion trajectory of the body part in the video frame sequence through the detection box sequence.

[0037] In some embodiments, such as Figure 3 As shown, step 220 includes steps 310-330, which are detailed below:

[0038] Step 310: For any two adjacent video frames in the video frame sequence, the first video frame and the second video frame, based on the position information of the detection boxes in the first video frame, the position information of the detection boxes in the second video frame, and the category corresponding to each detection box, determine the similarity between each detection box in the first video frame and each detection box in the second video frame.

[0039] In some embodiments, based on the category corresponding to each detection box, the positional overlap of different detection boxes belonging to the same category can be calculated, and the positional overlap between different detection boxes belonging to the same category can be used as the similarity between the two detection boxes. For different detection boxes belonging to different categories, they are set to be dissimilar, that is, no positional overlap calculation is required.

[0040] In a specific embodiment, the Intersection over Union (IOU) ratio between two detection boxes can be used as the degree of overlap between the two detection boxes. The IOU ratio between two detection boxes is the IOU ratio between the pixel regions enclosed by the two detection boxes. Typically, the detection boxes are rectangular. Based on the known position information of the detection boxes, each side of the detection box is determined accordingly. Therefore, for two detection boxes, the intersection and union regions of the areas enclosed by the two detection boxes can be determined based on their positions, and the IOU ratio between the two detection boxes can be determined based on the intersection and union regions of the pixel regions enclosed by the two detection boxes. Assume the i-th detection box in the first video frame is represented as... The j-th detection box in the second video frame is represented as Then the detection box With detection box The degree of overlap between the positions is S1:

[0041]

[0042] In some other embodiments, step 310 includes steps A1-A3, which are described in detail below:

[0043] Step A1: Calculate the positional overlap between each detection box in the first video frame and each detection box in the second video frame based on the positional information of the detection boxes in the first video frame and the second video frame.

[0044] It is understandable that a video frame may include multiple body parts of the same object; therefore, the object detection model may determine multiple detection boxes for a single video frame. Thus, in step 310, the positional overlap between each detection box in the first video frame and each detection box in the second video frame can be calculated. The method for calculating the positional overlap is described above and will not be repeated here.

[0045] Step A2: Determine the category similarity between each detection box in the first video frame and each detection box in the second video frame based on the category corresponding to each detection box in the first video frame and the category corresponding to each detection box in the second video frame.

[0046] Specifically, the categories corresponding to the detection boxes can be digitized into category vectors. These category vectors can be feature vectors reflecting the semantics of the category to which the detection box belongs. Then, the category similarity between the two detection boxes is calculated based on their respective category vectors. Specifically, the distance (cosine distance or Euclidean distance) between the two category vectors can be calculated, and the category similarity between the two detection boxes can be determined based on this distance. For example, the difference between 1 and the cosine distance between the two category vectors can be used as the category similarity between the two detection boxes. For example, the i-th detection box in the first video frame... With the j-th detection box in the second video frame The category similarity S2 between them can be expressed as:

[0047]

[0048] in, This represents the i-th detection box in the first video frame. The category vector of the corresponding category, This represents the j-th detection box in the second video frame. The category vector corresponding to the category, and R() represents the function that calculates the category similarity based on the category vector.

[0049] In other embodiments, for two detection boxes, if the two detection boxes correspond to the same category, the category similarity between the two detection boxes can be set to 1; conversely, if the two detection boxes correspond to different categories, the category similarity between the two detection boxes can be set to 0. Following a similar process, the category similarity between each detection box in the first video frame and each detection box in the second video frame can be determined.

[0050] Step A3: Determine the similarity between each detection box in the first video frame and each detection box in the second video frame based on the positional overlap between each detection box in the first video frame and each detection box in the second video frame, and the category similarity between each detection box in the first video frame and each detection box in the second video frame.

[0051] In some embodiments, the positional overlap between two detection boxes can be weighted by the category similarity, and the weighted result can be used as the similarity between the two detection boxes. For example, the i-th detection box in the first video frame With the j-th detection box in the second video frame similarity between It can be obtained using the following formula:

[0052]

[0053] Here, α1 and α2 are weighting coefficients, which can be set according to actual needs and are not specifically limited here.

[0054] In other embodiments, the positional overlap between two detection boxes can be multiplied by the category similarity, and the result can be used as the similarity between the two detection boxes. For example, the i-th detection box in the first video frame With the j-th detection box in the second video frame similarity between It can be obtained using the following formula:

[0055]

[0056] Step 320: Based on the similarity between each detection box in the first video frame and each detection box in the second video frame, determine the matching detection box pair, where one detection box in the detection box pair is located in the first video frame and the other detection box is located in the second video frame.

[0057] In a specific embodiment, it can be set that for each detection box (let's say detection box T) in the first video frame, only one detection box in the second video frame matches detection box T. Therefore, following steps 310-320 above, at most one detection box pair can be determined for each detection box in the first video frame in the second video frame.

[0058] In a specific embodiment, for any detection box in the first video frame (let's call it detection box T), the detection box in the second video frame with the highest similarity to the detection box T (let's call it detection box K) can be determined as the detection box that matches the detection box. Correspondingly, a detection box pair can be formed by detection box T and detection box K, namely {detection box T, detection box K}.

[0059] As described above, since a video frame may contain multiple detection boxes, for two adjacent video frames, assuming they have m and n detection boxes respectively, we can first calculate the similarity between each pair of detection boxes, forming an m×n similarity matrix. Then, we select the position with the largest value in the similarity matrix, let's say the coordinates are (i,j). We consider the i-th detection box of the preceding video frame to match the j-th detection box of the following video frame, forming a detection box pair. Next, we set all data in the i-th row and j-th column of the similarity matrix to 0 to ensure a one-to-one correspondence between detection boxes. Iterating through these steps determines all detection box pairs between any two adjacent video frames; unmatched detection boxes are retained for later use. Performing the same operation on all frames forms detection box pairs for the entire sequence.

[0060] Step 330: According to the temporal information of the video frame, combine the detection boxes in multiple detection box pairs of the same category to obtain multiple detection box sequences.

[0061] As determined by steps 310-320 above, the two detection boxes in a detection box pair belong to the same category. This means that if two detection boxes from two different video frames belong to the same category, they correspond to the same body part, such as the head. Based on this, combining the detection boxes from multiple detection box pairs of the same category yields a detection box sequence reflecting the motion of the same body part within the video frame sequence. Since the detection boxes are determined by the target detection model from the video frames, the temporal information of the video frame containing the detection box within the video frame sequence can be considered the temporal information corresponding to the detection box. Therefore, in step 330, according to the temporal information of the video frame containing the detection box within the video frame sequence and the order of the video frames in the video frame sequence, the detection boxes from multiple detection box pairs of the same category are combined to obtain the detection box sequence.

[0062] Typically, detection boxes in a video frame are rectangular. A sequence of detection boxes that links multiple detection boxes of the same category across multiple video frames visually appears as the trajectory of the corresponding body part, or as a pipe surrounding the body part. Figure 4 The diagram above illustrates a sequence of detection boxes.

[0063] Step 230: Based on the position information of the first detection box in each detection box sequence, the position information of the last detection box, and the timing information of the video frame containing the detection box in the video frame sequence, connect the different detection box sequences to obtain at least one combined detection box sequence.

[0064] Since the state of body parts in a video frame sequence is continuous, theoretically, the detection boxes for each body part in each video frame sequence are also continuous. That is, the number of detection boxes in the sequence for each body part is the same as the number of video frames in the video frame sequence. However, in practice, due to occlusion of body parts in video frames or blurring of video frames, the detection boxes detected by the target detection model corresponding to the body parts may not match the actual situation, i.e., there may be false detections. Alternatively, the target detection model may not detect the detection box corresponding to a certain body part in a certain video frame, i.e., there may be missed detections.

[0065] Therefore, in reality, the number of detection boxes in the detection box sequence obtained for each body part is less than the number of video frames in the video frame sequence. Furthermore, multiple detection box sequences may be obtained for the same body part, or a detection box sequence may contain only one detection box. In other words, a single detection box sequence may not fully reflect the motion of a body part within the video frame sequence. Therefore, in this application, after obtaining the detection box sequence, different detection box sequences are concatenated to obtain a combined detection box sequence spanning a longer timeline.

[0066] In some embodiments, such as Figure 5 As shown, step 230 includes steps 510-530, which are detailed below:

[0067] Step 510: For the two detection box sequences to be connected, based on the timing information of the video frames in each detection box sequence in the video frame sequence, determine the first detection box sequence that comes first in timing and the second detection box sequence that comes later in timing.

[0068] Understandably, before step 510, it is necessary to pre-determine which detection box sequences are to be connected. Specifically, for two detection box sequences, if the length of the two sequences is less than a length threshold, and the categories corresponding to the detection boxes in the two sequences are the same, then the two sequences can be determined to be detection box sequences that need to be connected. The duration of a detection box sequence can be represented by the number of detection boxes in the sequence.

[0069] In this application, for ease of distinction, the detection box sequence that comes first in the two detection box sequences to be connected is referred to as the first detection box sequence, and the detection box sequence that comes later in the time sequence is referred to as the second detection box sequence.

[0070] Specifically, for two detection box sequences to be connected (let's call them detection box sequence L1 and detection box sequence L2), the video frame containing the last detection box in detection box sequence L1 is video frame X1, and the video frame containing the first detection box in detection box sequence L2 is video frame X2. If, based on the timing information of the video frames containing the detection boxes in the video frame sequence, it is determined that the video frame containing the last detection box in detection box sequence L1 (i.e., video frame X1) is located before the video frame containing the first detection box in detection box sequence L2 (i.e., video frame X2) in the video frame sequence, then detection box sequence L1 can be determined as the first detection box sequence with a earlier timing, and detection box sequence L2 is the second detection box sequence with a later timing.

[0071] Step 520: Calculate the positional overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence based on the positional information of the last detection box in the first detection box sequence and the positional information of the first detection box in the second detection box sequence.

[0072] Similarly, the intersection-union ratio between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence can be used as the positional overlap between the two detection boxes.

[0073] Step 530: If the position overlap is greater than the overlap threshold, the first detection box sequence and the second detection box sequence are connected according to the temporal relationship between the first detection box sequence and the second detection box sequence to obtain a combined detection box sequence.

[0074] For two detection box sequences that correspond to the same body part, if the last detection box in the earlier detection box sequence has a high degree of positional overlap with the first detection box in the later detection box sequence, it indicates that the two detection box sequences are likely to be sequentially continuous. Therefore, in this application, when the positional overlap is greater than the overlap threshold, the first detection box sequence and the second detection box sequence are connected.

[0075] Since the two detection box sequences are connected according to the temporal relationship between them, the first detection box sequence is located before the second detection box sequence in the combined detection box sequence. In other words, the second detection box sequence is combined after the last detection box in the first detection box sequence, thus connecting the first and second detection box sequences.

[0076] In some embodiments, step 230 further includes: if the first video frame containing the last detection box in the first detection box sequence and the second video frame containing the first detection box in the second detection box sequence are not continuous in the video frame sequence, performing detection box completion on the combined detection box sequence so that the video frames containing any two adjacent detection boxes in the completed combined detection box sequence are continuous in the video frame sequence.

[0077] In this application, for ease of distinction, the video frame containing the last detection frame in the first detection frame sequence is referred to as the first video frame, and the video frame containing the first detection frame in the second detection frame sequence is referred to as the second video frame. For example, if the first video frame is determined to be the third video frame in the video frame sequence, and the second video frame is determined to be the fourth video frame in the video frame sequence, it is clear that the first video frame and the second video frame are not consecutive in the video frame sequence. If the first video frame containing the last detection frame in the first detection frame sequence and the second video frame containing the first detection frame in the second detection frame sequence are not consecutive in the video frame sequence, it indicates that there are missed or false detections for the body parts corresponding to the first detection frame sequence in the video frames between the first and second video frames. Therefore, in this case, the combined detection frame sequence can be completed, that is, the detection frames for the body parts corresponding to the first detection frame sequence can be completed in the video frames between the first and second video frames.

[0078] Within a short timeframe, the object's movements are continuous; therefore, the detection boxes corresponding to the same body part in multiple consecutive video frames are also generally continuous. Based on this, for the detection boxes to be completed, linear interpolation can be performed using the position information of the last detection box in the first detection box sequence and the position information of the first detection box in the second detection box sequence to determine the position information of the detection box corresponding to the body part in the first detection box sequence within the video frames between the first and second video frames. Then, the interpolated detection box position information can be added to the combined detection box sequence between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence.

[0079] In some embodiments, step 230 includes: for two detection box sequences, if the categories corresponding to the detection boxes in the two detection box sequences are the same, and the number of video frames between the last detection box in the first detection box sequence (which is earlier in the time sequence) and the first detection box in the second detection box sequence (which is later in the time sequence) is less than a threshold, then the first and second detection box sequences are connected according to their corresponding temporal relationships to obtain a combined detection box sequence. Correspondingly, if the number of video frames between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence (which is later in the time sequence) is not zero, then the combined detection box sequence is padded with detection boxes.

[0080] In some embodiments, in addition to outputting the position information of each detection box and the corresponding category of the detection box in the video frame, the object detection model further outputs the confidence score of each detection box. Based on this, the combined detection box sequence can be re-scored according to the confidence scores of each detection box in the combined detection box sequence. Specifically, the average confidence score of all detection boxes in the combined detection box sequence can be calculated, and the average of all confidence scores can be used as the confidence score of the combined detection box sequence; or, the maximum confidence score in the combined detection box sequence can be used as the confidence score of the combined detection box sequence. Currently, for the completed detection box, the confidence scores of the existing detection boxes used by the completed detection box can be used as the confidence score of the completed detection box. Of course, if the completed detection box uses multiple existing detection boxes, the average of the confidence scores of the multiple existing detection boxes used can be used as the confidence score of the completed detection box.

[0081] Based on the rescoring of the combined detection box sequence, non-maximum suppression can be further performed according to the confidence score corresponding to the combined detection box sequence, or combined detection box sequences that need to be used for keypoint detection can be selected according to the confidence score corresponding to the combined detection box sequence. For example, combined detection box sequences with confidence scores greater than the score threshold can be used for subsequent keypoint detection.

[0082] Step 240: For each combined detection box sequence, perform key point detection on the pixel region surrounded by each detection box in the combined detection box sequence to obtain the key point sequence corresponding to the combined detection box sequence.

[0083] In some embodiments, step 240 includes: performing keypoint detection on the pixel regions enclosed by each detection box in the combined detection box sequence using a keypoint detection model, and outputting the position information of each keypoint in the pixel regions enclosed by each detection box in the combined detection box sequence and the keypoint category corresponding to each keypoint; and combining the position information of keypoints belonging to the same keypoint category in the combined detection box sequence and in different video frames according to the keypoint category corresponding to the keypoints in each detection box in the combined detection box sequence to obtain a keypoint sequence corresponding to the combined detection box sequence.

[0084] Keypoint detection models can be constructed using one or more neural networks, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, etc., without specific limitations. In specific embodiments, a single keypoint detection model can be used to detect keypoints in multiple body parts (e.g., the entire body), or a separate keypoint detection model can be trained for each body part to detect keypoints in that body part.

[0085] To ensure the accuracy of the keypoint detection model, it needs to be trained using secondary training data. This secondary training data includes various secondary sample images and their annotation information. The annotation information of the secondary sample images indicates the location of each keypoint in the body parts within those images. The number of annotated keypoints in each body part can be determined based on actual needs. Annotated keypoints can be located at various organs (nose, left eye, right eye, left eyebrow, right eyebrow, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle, etc.). Of course, multiple keypoints can be annotated at the same organ. Figure 6 An illustrative diagram of key points on the face is shown. Figure 6 As shown, there are three key points for both the left and right eyebrows, three key points for both the left and right eyes, and three key points for the nose and mouth. It is worth mentioning that... Figure 6 This is merely an example; in a specific embodiment, the number of key points for facial annotation may be more or less.

[0086] During training, a second sample image is input into the keypoint detection model. The keypoint detection model extracts image features from the second sample image and predicts the location information of keypoints in the second sample image (for ease of distinction, the location information of keypoints output by the keypoint detection model for the second sample image is called predicted location information) and the corresponding keypoint category (for ease of distinction, the keypoint category corresponding to the keypoint output by the keypoint detection model for the first sample image is called predicted keypoint category). In this application, the keypoint category corresponding to a keypoint is used to uniquely indicate that keypoint; in other words, one keypoint category corresponds to one keypoint, for example... Figure 6 Among the facial key points shown, the leftmost key point on the upper left eye is considered as one key point category, the middle key point on the upper left eye is considered as one key point category, and the rightmost key point on the upper left eye is considered as one key point category.

[0087] Subsequently, based on the annotation information corresponding to the second sample image, and the predicted location information and corresponding keypoint category of each keypoint output by the keypoint detection model for the second sample image, the loss value of the loss function is calculated. The parameters of the keypoint detection model are then adjusted in reverse based on the loss value until the second training termination condition is met. This second termination condition can be that the number of iterations of the keypoint detection model reaches a second threshold, or that the loss function corresponding to the keypoint detection model converges; no specific limitation is made here. The loss function corresponding to the keypoint detection model can be a cross-entropy loss function, a mean squared error loss function, a smoothed mean absolute error loss function, etc., without specific limitation. After training, the keypoint detection model is used for keypoint detection to ensure the accuracy of the keypoint detection performed by the keypoint detection model.

[0088] Based on the trained keypoint detection model, the pixel regions enclosed by each detection box in the combined detection box sequence can be input into the keypoint detection model. The keypoint detection model outputs the position information of keypoints in the pixel regions enclosed by each detection box and the corresponding keypoint category. Then, according to the temporal order of the video frames containing the keypoints in the video frame sequence, keypoints belonging to the same keypoint category and appearing in multiple video frames are combined to obtain the corresponding keypoint sequence. It can be understood that the keypoint sequence includes the keypoints in the pixel regions enclosed by each detection box in the combined detection box sequence.

[0089] It's worth noting that when the keypoint detection model is used to detect keypoints in multiple body parts, it can also associate the combined detection box sequences of multiple body parts belonging to the same object based on the combined detection box sequences corresponding to different body parts, thus obtaining an object detection box sequence. In other words, the object detection box sequence includes the detection boxes corresponding to each body part of the same object in each video frame. Alternatively, by utilizing the spatiotemporal relationship between different combined detection box sequences, the detection boxes corresponding to multiple body parts on the same object are connected, thereby forming an object detection box sequence for the same object, where each object includes body parts such as the body, head, left hand, and right hand. Based on this, the pixel region enclosed by multiple detection boxes in the same video frame from the object detection box sequence is input into the keypoint detection model for keypoint detection.

[0090] In some embodiments, after processing according to steps 220-230 above, there may still be short detection box sequences or combined detection box sequences. Since short detection box sequences or combined detection box sequences cannot fully reflect the motion in the video frame sequence, a length threshold can be set. After step 230, combined detection box sequences with a length less than the length threshold do not need to undergo subsequent key point detection steps.

[0091] In this application, based on body part detection performed on each video frame in a video frame sequence to obtain the position information of the detection boxes and the corresponding categories of each detection box, multiple detection box sequences are determined according to the position information of the detection boxes in different video frames in the video frame sequence and the corresponding categories of each detection box. Then, based on the position information of the first detection box, the position information of the last detection box, and the temporal information of the video frame in which the detection box is located in the video frame sequence, the different detection box sequences are connected to obtain at least one combined detection box sequence. After that, key point detection is performed on the pixel regions surrounded by each detection box in the combined detection box sequence to obtain the key point sequence corresponding to the combined detection box sequence. To fully consider the potential for missed or false detections during body part detection, which could lead to the resulting detection box sequence failing to accurately track the movement of body parts within a video frame sequence, this paper combines the position information of the first and last detection boxes in each detection box sequence, along with the temporal information of the video frame containing the detection box within the video frame sequence, to connect different detection box sequences. This ensures that the resulting combined detection box sequence can accurately track the movement of body parts within the video frame sequence, thereby guaranteeing that the resulting keypoint sequence accurately reflects the actions of each body part within the video frame sequence and ensuring the correspondence between the obtained keypoint sequence and the video frame sequence.

[0092] In some embodiments, after step 240, such as Figure 7 As shown, the method also includes:

[0093] Step 710: Identify abnormal key points in the key point sequence.

[0094] In practice, the clarity of video frames may cause keypoint detection models to falsely detect or miss keypoints, leading to abnormal jumps in keypoint sequences. Therefore, to address this issue, this application further repairs abnormal keypoints in the keypoint sequence, allowing the repaired keypoint sequence to be used in subsequent applications.

[0095] In some embodiments, step 710 includes: determining key points with confidence levels lower than a second confidence threshold as abnormal key points based on the confidence levels of each key point in the key point sequence; and / or determining motion parameters of each key point in the key point sequence based on the position information of each key point in the key point sequence, and determining key points with motion parameters greater than a motion parameter threshold as abnormal key points, wherein the motion parameters include at least one of velocity and acceleration.

[0096] In some embodiments, the keypoint detection model further predicts and outputs the confidence level of each keypoint. Based on this, keypoints with a confidence level lower than a second confidence threshold are considered abnormal keypoints. The second confidence threshold can be set according to actual needs and is not specifically limited here.

[0097] In a keypoint sequence, if the velocity or acceleration of a keypoint is large, it indicates a jump in the keypoint. Therefore, in some embodiments, keypoints in the keypoint sequence whose velocity or acceleration exceeds the corresponding motion parameter threshold are considered abnormal keypoints. In some embodiments, a velocity threshold can be set for velocity and an acceleration threshold can be set for acceleration (i.e., the motion parameter thresholds include velocity thresholds and acceleration thresholds), and keypoints in the keypoint sequence whose velocity exceeds the velocity threshold and / or whose acceleration exceeds the acceleration threshold are identified as abnormal keypoints.

[0098] In a specific embodiment, since the keypoint sequence includes multiple keypoints of the same keypoint category located on different video frames, and the keypoint detection model outputs the position information of each keypoint, for multiple keypoints belonging to the same keypoint category and being temporally adjacent in the keypoint sequence, the velocity of each keypoint can be determined by the position information of the keypoint and the time interval between adjacent video frames. Then, the acceleration of each keypoint can be determined based on the velocity of each keypoint and the time interval between adjacent video frames.

[0099] Step 720: Determine the target video frame among the video frames adjacent to the video frame where the abnormal key point is located.

[0100] In some embodiments, the keypoint detection model further outputs the confidence score of each keypoint in the pixel region enclosed by each detection box. Based on this, according to the confidence scores of keypoints of the same category as the keypoint indicated by the abnormal keypoint in video frames adjacent to the video frame containing the abnormal keypoint, a third video frame containing a keypoint of the same category as the keypoint indicated by the abnormal keypoint and with a confidence score greater than a first confidence threshold can be determined from the video frames adjacent to the video frame containing the abnormal keypoint. This third video frame is then used as the target video frame. Video frames adjacent to the video frame containing the abnormal keypoint can be video frames whose time interval with the video frame containing the abnormal keypoint is less than a time threshold. The time threshold can be set according to actual needs and is not specifically limited here.

[0101] In some embodiments, due to issues with video frame quality or model robustness, sudden jumps in keypoint confidence scores may occur, potentially impacting subsequent judgments. Therefore, a re-scoring process is performed before subsequent steps. This step does not alter the overall trend of keypoint confidence scores. Specifically, a one-dimensional mean filter is applied to the confidence scores of each keypoint sequence to smooth out some spikes. Then, the target video frame is determined based on the re-scored keypoint confidence scores.

[0102] In other embodiments, the two video frames adjacent to the video frame containing the abnormal key point are identified as the target video frame.

[0103] Step 730: Repair the abnormal keypoints based on the location information of keypoints in the target video frame that belong to the same keypoint category as the abnormal keypoints.

[0104] In other words, the location information of the abnormal keypoint is redefined by interpolating the location information of keypoints in the target video frame that belong to the same keypoint category as the abnormal keypoint. In a specific embodiment, linear interpolation can be performed using the location information of keypoints in the target video frame that belong to the same keypoint category as the abnormal keypoint to redefine the location information of the abnormal keypoint, and the location information of the abnormal keypoint in the keypoint sequence can be updated accordingly with the redefined location information.

[0105] In this embodiment, spatiotemporal information in the keypoint sequence is used to repair abnormal keypoints, thereby reducing the occurrence of keypoint jumps in the keypoint sequence.

[0106] In some embodiments, the body part includes the feet; the keypoint sequence includes foot keypoints in each video frame; in this embodiment, the method further includes the following steps A1-A2:

[0107] Step A1: Predict the foot landing state based on the foot key points in the key point sequence to obtain the foot prediction result.

[0108] In a specific embodiment, a foot landing state can be predicted based on key foot points using a foot prediction model. The foot prediction results are used to indicate the area of ​​the foot in contact with the ground, such as left toe landing (i.e., the left toe area is in contact with the ground), left heel landing, right toe landing, right heel landing, etc.

[0109] Footstep prediction models can be constructed using one or more neural networks, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, etc., without specific limitations here.

[0110] To ensure the accuracy of the foot landing prediction model in predicting foot landing states, it needs to be trained in advance using third training data. This third training data includes keypoints in a third sample image and corresponding labeled foot landing information. The labeled foot landing information indicates the landing area of ​​the foot in the third sample image. In some embodiments, the labeled foot landing information can be a four-dimensional vector, where each dimension represents whether the left toe, left heel, right toe, and right heel are on the ground. In some embodiments, the keypoints in the third sample image can include full-body keypoints or only keypoints of the lower body, but must include at least keypoints of the heel and toes.

[0111] During training, key points from the third sample image are input into the foot prediction model. The model predicts the foot landing state based on these key points and outputs the predicted foot landing result for the third sample image. Then, the loss function is calculated based on the predicted foot landing result and the labeled foot landing information from the third sample image. The parameters of the foot prediction model are adjusted in reverse based on the loss value until the third training termination condition is met. The third training termination condition can be either reaching the third iteration threshold or the convergence of the loss function. The loss function of the foot prediction model can be a cross-entropy loss function, a mean squared error loss function, a smoothed mean absolute error loss function, etc., without specific limitations.

[0112] Step A2 repairs abnormal foot key points in the key point sequence based on the foot prediction results.

[0113] Specifically, based on the foot prediction results, the key points of the foot that landed in the corresponding video frame can be determined. If the abnormal foot key point is not the determined foot key point, the foot key point that landed can be fixed in a similar way as described above. The target video frame is determined for the abnormal foot key point, and the position information of the abnormal foot key point is re-determined by linear interpolation based on the position information of the key points in the target video frame that belong to the same key point category as the abnormal foot key point and the position information of the foot key point that landed in the target video frame.

[0114] In other embodiments, if the abnormal foot key point is a foot key point that is determined to be on the ground, since the change in the foot's ground state is small in a short period of time, in other words, the position of the foot key point is fixed in a short period of time, in this case, for the target video frame where the abnormal foot key point is determined, the position information of key points in the target video frame that belong to the same key point category as the abnormal foot key point can be determined as the position information of the abnormal foot key point, thereby realizing the repair of the abnormal foot key point.

[0115] In some embodiments, the foot prediction results output by the foot prediction model may have some jitter. Therefore, a preset kernel is used to filter the foot prediction results to obtain a smoother foot state. The filtering kernel includes, but is not limited to, the following forms:

[0116] [0,0,1,0,0],[1,1,0,1,1],[0,0,0,1,1,0,0,0].

[0117] By combining the foot prediction results with the confidence of key points on the foot, the key points of the foot can be repaired. Since the position of the center of gravity has been determined, the key points of a certain position on the foot can be fixed for a period of time. Even if there are subtitles or other obstructions, foot shaking can be prevented to a certain extent.

[0118] In this embodiment, combining the foot prediction results to repair abnormal foot key points can still obtain a smooth and reasonable key point sequence even when there is text or special effects occlusion in the video frame, thus avoiding the problem of foot instability caused by subtitle occlusion.

[0119] In some embodiments, after step 240, the method further includes smoothing the keypoint sequence. This allows the smoothed keypoint sequence to be used in subsequent applications.

[0120] In a specific embodiment, a keypoint sequence smoothing model can be used to smooth the keypoint sequence. This keypoint sequence smoothing model is a model constructed using one or more neural networks, such as convolutional neural networks, recurrent neural networks, fully connected neural networks, etc., without specific limitations.

[0121] Before applying the keypoint sequence smoothing model online, it is trained using a fourth training dataset. This fourth training dataset includes sample keypoint sequences and their corresponding smoothed keypoint sequences. The smoothed keypoint sequences indicate the new positional information of each keypoint in the sample keypoint sequence after smoothing. Then, the sample keypoint sequences are input into the keypoint sequence smoothing model, which smooths them and outputs the smoothed result. This result indicates the redefined positional information of each keypoint in the sample keypoint sequence. Then, based on the location information of each key point indicated by the smoothing result corresponding to the sample key point sequence and the location information of each key point indicated by the smoothed key point sequence corresponding to the sample key point sequence, the loss value of the loss function corresponding to the key point sequence smoothing model is calculated, and the parameters of the key point sequence smoothing model are adjusted in reverse according to the loss value until the fourth training termination condition is reached. The fourth training termination condition can be that the number of iterations of the key point sequence smoothing model reaches the fourth threshold, or that the loss function corresponding to the key point sequence smoothing model converges.

[0122] In some embodiments, the loss function corresponding to the keypoint sequence smoothing model can be:

[0123]

[0124] In Formula 5 above, the first term is the position loss, and the second term is the acceleration loss. Here, T is the total number of frames or time points, C is the total number of channels (i.e., the dimension to be smoothed, or the total number of keypoint categories), and W is the weight. Let Y be the predicted position of the c-th dimension in frame t. c, Let c be the true position of the c-th dimension in frame t (i.e., the position indicated by the smoothed keypoint sequence). Let A be the predicted acceleration of the c-th dimension in frame t (i.e., the acceleration calculated from the positions of keypoints indicated by the smoothing process). c, W represents the true acceleration of the c-th dimension in frame t (i.e., the acceleration of the keypoints calculated from the positions indicated by the smoothed keypoint sequence). pose W is a constant set for position loss. acc It is a constant set to account for acceleration loss.

[0125] In some embodiments, the keypoint sequence may be repaired first, and then smoothed. This ensures that the final keypoint sequence is logical and smooth overall, reducing jumps in keypoints within the sequence.

[0126] In some embodiments, after step 240, the method further includes controlling the virtual object to perform actions according to a sequence of key points.

[0127] Based on keypoint sequences, action recognition can be performed, thereby allowing virtual objects to perform actions corresponding to the keypoint sequences. This enables the control of virtual object actions through the actions of real people.

[0128] The following describes an apparatus embodiment of this application, which can be used to perform the methods described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the method embodiments described in the above embodiments of this application.

[0129] Figure 8 This is a block diagram of a key point detection device according to an embodiment of this application. This key point detection device can be configured in an electronic device to implement the key point detection method provided in this application. Figure 8 As shown, the keypoint detection device includes: a detection module 810, used to detect body parts in each video frame of a video frame sequence, obtaining the position information of the detection boxes in each video frame and the category corresponding to each detection box, wherein the category corresponding to the detection box is used to indicate the body part to which the pixel area surrounded by the detection box belongs; a detection box sequence determination module 820, used to determine multiple detection box sequences based on the position information of the detection boxes in different video frames of the video frame sequence and the category corresponding to each detection box, wherein the pixel areas surrounded by different detection boxes in a detection box sequence belong to the same body part; a connection module 830, used to connect different detection box sequences based on the position information of the first detection box in each detection box sequence, the position information of the last detection box, and the temporal information of the video frame in the video frame sequence where the detection box is located, to obtain at least one combined detection box sequence; and a keypoint detection module 840, used to perform keypoint detection on the pixel areas surrounded by each detection box in each combined detection box sequence, to obtain a keypoint sequence corresponding to the combined detection box sequence.

[0130] In some embodiments, the connection module 830 includes: a timing determination unit, configured to, for two detection box sequences to be connected, determine a first detection box sequence that precedes the second detection box sequence based on the timing information of the video frames containing the detection boxes in each of the two detection box sequences in the video frame sequence; a position overlap calculation unit, configured to calculate the position overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence based on the position information of the last detection box in the first detection box sequence and the position information of the first detection box in the second detection box sequence; and a connection unit, configured to, if the position overlap is greater than an overlap threshold, connect the first detection box sequence and the second detection box sequence according to the timing relationship between the two detection box sequences to obtain a combined detection box sequence.

[0131] In some embodiments, the connection module 830 further includes: a completion unit, configured to perform detection box completion on the combined detection box sequence if the first video frame containing the last detection box in the first detection box sequence and the second video frame containing the first detection box in the second detection box sequence are not continuous in the video frame sequence, so that the video frames containing any two adjacent detection boxes in the completed combined detection box sequence are continuous in the video frame sequence.

[0132] In some embodiments, the detection box sequence determination module 820 includes: a similarity determination unit, configured to determine the similarity between each detection box in the first video frame and each detection box in the second video frame for any two adjacent video frames in the video frame sequence, based on the position information of the detection boxes in the first video frame, the position information of the detection boxes in the second video frame, and the category corresponding to each detection box; a detection box pair determination unit, configured to determine matching detection box pairs based on the similarity between each detection box in the first video frame and each detection box in the second video frame, wherein one detection box in the detection box pair is located in the first video frame and the other detection box is located in the second video frame; and a first combination unit, configured to combine the detection boxes in multiple detection box pairs belonging to the same category according to the temporal information of the video frames to obtain multiple detection box sequences.

[0133] In some embodiments, the similarity determination unit includes: a first calculation unit, configured to calculate the positional overlap between each detection box in the first video frame and each detection box in the second video frame based on the positional information of the detection boxes in the first video frame and the positional information of the detection boxes in the second video frame; a category similarity determination unit, configured to determine the category similarity between each detection box in the first video frame and each detection box in the second video frame based on the category corresponding to each detection box in the first video frame and the category corresponding to each detection box in the second video frame; and a first determination unit, configured to determine the similarity between each detection box in the first video frame and each detection box in the second video frame based on the positional overlap between each detection box in the first video frame and each detection box in the second video frame, and the category similarity between each detection box in the first video frame and each detection box in the second video frame.

[0134] In some embodiments, the keypoint detection module 840 includes: a keypoint detection unit, configured to perform keypoint detection on the pixel regions enclosed by each detection box in the combined detection box sequence using a keypoint detection model, and output the position information of each keypoint in the pixel regions enclosed by each detection box in the combined detection box sequence and the keypoint category corresponding to each keypoint; and a second combination unit, configured to combine the position information of keypoints belonging to the same keypoint category in the combined detection box sequence and in different video frames according to the keypoint category corresponding to the keypoints in each detection box in the combined detection box sequence, to obtain a keypoint sequence corresponding to the combined detection box sequence.

[0135] In some embodiments, the key point detection device further includes: an abnormal key point determination unit, configured to determine abnormal key points in the key point sequence; a target video frame determination unit, configured to determine a target video frame in video frames adjacent to the video frame where the abnormal key point is located; and a repair unit, configured to repair the abnormal key point based on the location information of key points in the target video frame that belong to the same key point category as the abnormal key point.

[0136] In some embodiments, the keypoint detection model further outputs the confidence level of each keypoint in the pixel region enclosed by each detection box; the target video frame determination unit includes: determining, based on the confidence level of keypoints in video frames adjacent to the video frame where the abnormal keypoint is located that have the same keypoint category as the keypoint indicated by the abnormal keypoint, a third video frame containing a keypoint that has the same keypoint category as the keypoint indicated by the abnormal keypoint and a confidence level greater than a first confidence threshold, and using the third video frame as the target video frame; and / or determining two video frames adjacent to the video frame where the abnormal keypoint is located as the target video frames.

[0137] In some embodiments, the abnormal key point determination unit includes: determining key points with a confidence level lower than a second confidence level threshold as abnormal key points based on the confidence level of each key point in the key point sequence; and / or determining the motion parameters of each key point in the key point sequence based on the position information of each key point in the key point sequence, and determining key points with motion parameters greater than a motion parameter threshold as abnormal key points, wherein the motion parameters include at least one of velocity and acceleration.

[0138] In some embodiments, the body part includes the foot; the key point sequence includes foot key points in each video frame; the key point detection device further includes: a foot landing state prediction module, used to predict the foot landing state based on the foot key points in the key point sequence, and obtain a foot prediction result; and a second repair module, used to repair abnormal foot key points in the key point sequence based on the foot prediction result.

[0139] In some embodiments, the key point detection device further includes a smoothing module for smoothing the key point sequence.

[0140] In some embodiments, the key point detection device further includes: an action control module for controlling the virtual object to perform actions according to the key point sequence.

[0141] Figure 9 This is a schematic diagram of an electronic device according to an embodiment of this application. The electronic device is used to perform the key point detection method provided in this application.

[0142] like Figure 9 As shown, the electronic device may include: a processor 901, such as a CPU; a network interface 904; a user interface 903; a memory 905; and a communication bus 902. The communication bus 902 is used to enable communication between these components. The user interface 903 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 903 may also include a standard wired interface or a wireless interface. The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 905 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 905 may also be a storage device independent of the aforementioned processor 901.

[0143] Those skilled in the art will understand that Figure 9 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Figure 9As shown, the memory 905, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a program for implementing the key point detection method.

[0144] exist Figure 9 In the illustrated electronic device, the network interface 904 is primarily used for communication connections with other devices, such as... Figure 1 Terminal 110, etc. The user interface 903 is mainly used to connect to the client (user terminal) and communicate with the client; while the processor 901 can be used to call the program implementing the key point detection method stored in the memory 905 and execute the steps of the key point detection method as in any of the above method embodiments.

[0145] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, on which computer-readable instructions are stored, which, when executed by a processor, implement the key point detection method as described in any of the above method embodiments.

[0146] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0147] According to one aspect of the embodiments of this application, a computer program product is provided, which includes computer instructions that, when executed by a processor, implement the key point detection method as described in any of the above method embodiments.

[0148] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0149] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0150] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0151] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0152] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A key point detection method, characterized in that, include: Body part detection is performed on each video frame in the video frame sequence to obtain the position information of the detection box in each video frame and the category corresponding to each detection box. The category corresponding to the detection box is used to indicate the body part to which the pixel area surrounded by the detection box belongs. Based on the position information of the detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box, multiple detection box sequences are determined, wherein the pixel regions surrounded by different detection boxes in a detection box sequence belong to the same body part. Based on the position information of the first detection box and the position information of the last detection box in each detection box sequence, and the timing information of the video frame containing the detection box in the detection box sequence within the video frame sequence, different detection box sequences are connected to obtain at least one combined detection box sequence; wherein, for the first detection box sequence and the second detection box sequence to be connected, if the positional overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence is greater than the overlap threshold, the first detection box sequence and the second detection box sequence are connected according to the timing relationship between the first detection box sequence and the second detection box sequence to obtain the combined detection box sequence; For each combined detection box sequence, key point detection is performed on the pixel region surrounded by each detection box in the combined detection box sequence to obtain the key point sequence corresponding to the combined detection box sequence.

2. The method according to claim 1, characterized in that, Before concatenating the first and second detection box sequences according to their temporal relationship if the overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence is greater than an overlap threshold, the method further includes: For two detection box sequences to be connected, the first detection box sequence that comes first and the second detection box sequence that comes later are determined based on the timing information of the video frames in the video frame sequence in each of the two detection box sequences. Based on the position information of the last detection box in the first detection box sequence and the position information of the first detection box in the second detection box sequence, the position overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence is calculated.

3. The method according to claim 2, characterized in that, The step of connecting different detection box sequences to obtain at least one combined detection box sequence based on the position information of the first detection box in each detection box sequence, the position information of the last detection box, and the timing information of the video frame containing the detection box in the video frame sequence, further includes: If the last detection box in the first detection box sequence is located in the first video frame and the first detection box in the second detection box sequence is located in the second video frame, which are not continuous in the video frame sequence, the combined detection box sequence is padded with detection boxes so that any two adjacent detection boxes in the padded combined detection box sequence are continuous in the video frame sequence.

4. The method according to claim 1, characterized in that, The step of determining multiple detection box sequences based on the position information of detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box includes: For any two adjacent video frames in a video frame sequence, the first video frame and the second video frame are used to determine the similarity between each detection box in the first video frame and each detection box in the second video frame based on the position information of the detection box in the first video frame, the position information of the detection box in the second video frame, and the category corresponding to each detection box. Based on the similarity between each detection box in the first video frame and each detection box in the second video frame, a matching detection box pair is determined, wherein one detection box in the detection box pair is located in the first video frame and the other detection box is located in the second video frame; Based on the temporal information of the video frame, multiple detection boxes belonging to the same category are combined to obtain multiple detection box sequences.

5. The method according to claim 4, characterized in that, The step of determining the similarity between each detection box in the first video frame and each detection box in the second video frame based on the position information of the detection boxes in the first video frame, the position information of the detection boxes in the second video frame, and the category corresponding to each detection box includes: Based on the position information of the detection boxes in the first video frame and the position information of the detection boxes in the second video frame, calculate the position overlap between each detection box in the first video frame and each detection box in the second video frame. Based on the categories corresponding to each detection box in the first video frame and the categories corresponding to each detection box in the second video frame, determine the category similarity between each detection box in the first video frame and each detection box in the second video frame; The similarity between each detection box in the first video frame and each detection box in the second video frame is determined based on the positional overlap between each detection box in the first video frame and each detection box in the second video frame, and the category similarity between each detection box in the first video frame and each detection box in the second video frame.

6. The method according to claim 1, characterized in that, For each combined detection box sequence, key point detection is performed on the pixel region enclosed by each detection box in the combined detection box sequence to obtain the key point sequence corresponding to the combined detection box sequence, including: The key point detection model is used to detect key points in the pixel regions surrounded by each detection box in the combined detection box sequence, and outputs the position information of each key point in the pixel regions surrounded by each detection box in the combined detection box sequence and the key point category corresponding to each key point. Based on the key point category corresponding to the key point in each detection box in the combined detection box sequence, the position information of key points belonging to the same key point category in different video frames in the combined detection box sequence is combined to obtain the key point sequence corresponding to the combined detection box sequence.

7. The method according to claim 6, characterized in that, After combining the position information of key points belonging to the same key point category in different video frames in the combined detection box sequence according to the key point category of each detection box in the combined detection box sequence to obtain the key point sequence corresponding to the combined detection box sequence, the method further includes: Identify the abnormal key points in the key point sequence; Determine the target video frame from the video frames adjacent to the video frame containing the abnormal key point; The abnormal key point is repaired based on the location information of key points in the target video frame that belong to the same key point category as the abnormal key point.

8. The method according to claim 7, characterized in that, The key point detection model also outputs the confidence level of each key point in the pixel region surrounded by each detection box; Determining the target video frame among video frames adjacent to the video frame containing the abnormal key point includes: Based on the confidence level of key points in video frames adjacent to the video frame where the abnormal key point is located that are of the same category as the key point indicated by the abnormal key point, a third video frame containing a key point that is of the same category as the key point indicated by the abnormal key point and has a confidence level greater than a first confidence threshold is determined in the video frames adjacent to the video frame where the abnormal key point is located, and the third video frame is used as the target video frame. And / or, The two video frames adjacent to the video frame containing the abnormal key point are identified as the target video frames.

9. The method according to claim 7, characterized in that, The process of determining abnormal key points in the key point sequence includes: Based on the confidence level of each key point in the key point sequence, key points with a confidence level lower than the second confidence threshold are identified as abnormal key points; And / or, Based on the position information of each key point in the key point sequence, the motion parameters of each key point in the key point sequence are determined, and key points whose motion parameters are greater than the motion parameter threshold are identified as abnormal key points. The motion parameters include at least one of velocity and acceleration.

10. The method according to claim 1, characterized in that, The body parts include the feet; the key point sequence includes key points of the feet in each video frame; The method further includes: Foot landing state is predicted based on the foot key points in the key point sequence to obtain foot prediction results; Based on the footstep prediction results, abnormal foot key points in the key point sequence are repaired.

11. The method according to claim 1, characterized in that, After performing keypoint detection on the pixel regions enclosed by each detection box in the combined detection box sequence to obtain the keypoint sequence corresponding to the combined detection box sequence, the method further includes: The key point sequence is smoothed.

12. The method according to any one of claims 1 to 11, characterized in that, After performing keypoint detection on the pixel regions enclosed by each detection box in the combined detection box sequence to obtain the keypoint sequence corresponding to the combined detection box sequence, the method further includes: Control the virtual object to perform actions according to the sequence of key points.

13. A key point detection device, characterized in that, include: The detection module is used to detect body parts in each video frame in the video frame sequence, and obtain the position information of the detection box in each video frame and the category corresponding to each detection box. The category corresponding to the detection box is used to indicate the body part to which the pixel area surrounded by the detection box belongs. The detection box sequence determination module is used to determine multiple detection box sequences based on the position information of the detection boxes in different video frames in the video frame sequence and the category corresponding to each detection box, wherein the pixel regions surrounded by different detection boxes in a detection box sequence belong to the same body part. A connection module is used to connect different detection box sequences based on the position information of the first detection box in each detection box sequence, the position information of the last detection box in each detection box sequence, and the timing information of the video frame containing the detection box in the detection box sequence in the video frame sequence, to obtain at least one combined detection box sequence; wherein, for the first detection box sequence and the second detection box sequence to be connected, if the positional overlap between the last detection box in the first detection box sequence and the first detection box in the second detection box sequence is greater than the overlap threshold, the first detection box sequence and the second detection box sequence are connected according to the timing relationship between the first detection box sequence and the second detection box sequence to obtain the combined detection box sequence; The key point detection module is used to perform key point detection on the pixel regions surrounded by each detection box in the combined detection box sequence for each combined detection box sequence, so as to obtain the key point sequence corresponding to the combined detection box sequence.

14. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium storing computer-readable instructions thereon, characterized in that, When the computer-readable instructions are executed by a processor, the method as described in any one of claims 1 to 12 is implemented.

Images