Target object key point detection method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a target object key point detection method and device, electronic equipment and storage medium. The method comprises: obtaining the similarity between adjacent first and second video frames; when the similarity is less than a preset threshold, calling an object detection model to determine the target object region in the second video frame; when the similarity is greater than or equal to the preset threshold, determining the target object region in the second video frame based on the target object key points detected in the first video frame, and then detecting the target object key points in the second video frame based on the target object region. The present disclosure can determine the target object region through the object detection model when the similarity is low, avoid frequent calling, reduce terminal power consumption, and determine the target object region based on the key points of the previous video frame when the similarity is high, avoid the wrong use of the key points of the previous frame, effectively improve the recognition accuracy, and achieve a balance between terminal power consumption and key point recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a target object key point detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of computer technology, human key point detection technology as a bottom algorithm support plays an increasingly important role. Taking video editing as an example, different parts of the human body in the video are recognized through human key point detection technology, and a user can beautify the specified human body part in the video, such as adjusting the thickness of the legs or waist, thereby providing better video effects.

[0003] In the related art, for human beautification in the video editing process, human key points in each video frame can be recognized one by one, or video frames can be selected according to a preset interval, and human key points are recognized.

[0004] However, in the above method, the former requires a high computing capability of a processing device, and the latter can reduce the amount of calculation to a certain extent, but there is a problem of inaccurate human key point detection. It can be seen that the related art has a problem that it is difficult to balance the recognition accuracy and the performance of the processing device when recognizing human key points. SUMMARY

[0005] The present disclosure provides a target object key point detection method and device, electronic equipment and storage medium to at least solve the problem that it is difficult to balance the recognition accuracy and the performance of the processing device when recognizing key points in the related art. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a target object key point detection method is provided, comprising:

[0007] obtaining a similarity between adjacent first and second video frames; the video frame order of the first video frame is before the second video frame, and the second video frame is a current video frame to be detected;

[0008] when the similarity is less than a preset threshold, calling an object detection model to determine a target object region in the second video frame;

[0009] when the similarity is greater than or equal to the preset threshold, determining a target object region in the second video frame based on the detected target object key points of the first video frame;

[0010] detecting target object key points in the second video frame based on the target object region determined from the second video frame.

[0011] In an example embodiment, the detecting the target object key points in the second video frame based on the target object region determined from the second video frame comprises:

[0012] inputting an image corresponding to the target object region into a key point detection model to detect a probability of each pixel point in the image being a target object key point by the key point detection model, and determining the target object key points in the image based on the probability.

[0013] In an example embodiment, the key point detection model is obtained based on the following manner:

[0014] obtaining a training image containing a target object and real key points corresponding to the target object in the training image; the real key points include first type key points and second type key points; the first type key points include key points of limb parts, and the second type key points include key points of trunk parts;

[0015] inputting the training image into a neural network model to be trained to identify a probability of each pixel point in the training image being a target object key point by the neural network model, and determining a plurality of predicted key points based on the probability;

[0016] determining a loss function of the neural network model at present by a preset supervision module in the neural network model; wherein the supervision module determines the loss function based on a first loss function corresponding to the real key points and the predicted key points, and a second loss function corresponding to a reference vector and a current vector; the reference vector is a vector representing relative positions between the first type key points and the second type key points in the real key points, and the current vector is a vector representing relative positions between predicted first type key points and predicted second type key points in the plurality of predicted key points;

[0017] adjusting parameters of the neural network model based on the loss function until a training end condition is met, to obtain a trained neural network model;

[0018] deleting the supervision module from the trained neural network model to obtain the key point detection model.

[0019] In an example embodiment, the neural network model includes a plurality of network modules connected in series, and the network modules are configured to output corresponding heat maps for input images; a response value corresponding to each pixel point in the heat map represents a probability of the pixel point being a target object key point.

[0020] The training image is input into the neural network model to be trained to identify a probability of each pixel point in the training image being a key point of the target object through the neural network model, and the method comprises the following steps of:

[0021] The feature map corresponding to the training image is taken as an input image of a first network module in the neural network model, and a heat map output by a previous network module is taken as an input image of a subsequent network module;

[0022] Based on a heat map output by a last network module, a probability of each pixel point in the training image being a key point of the target object is determined.

[0023] In an exemplary embodiment, each network module is provided with a supervision module;

[0024] The current loss function of the neural network model is determined based on the reference vector and the current vector through the preset supervision module, and the method comprises the following steps of:

[0025] For each network module, a current vector corresponding to a plurality of predicted key points in a heat map output by the network module is determined, and a local loss function corresponding to the network module is determined based on a reference vector and the current vector corresponding to the network module through the supervision module corresponding to the network module;

[0026] The loss function of the neural network model is determined based on the local loss functions corresponding to the plurality of network modules.

[0027] In an exemplary embodiment, the similarity between the adjacent first video frame and the second video frame is obtained, and the method comprises the following steps of:

[0028] A first gray-scale image corresponding to the first video frame is obtained, and first distribution information corresponding to pixel values in the first video frame is determined based on the first gray-scale image;

[0029] A second gray-scale image corresponding to the second video frame is obtained, and second distribution information corresponding to pixel values in the second video frame is determined based on the second gray-scale image;

[0030] The similarity between the first video frame and the second video frame is determined according to the similarity of the first distribution information and the second distribution information.

[0031] In an exemplary embodiment, the target object region in the second video frame is determined based on the target object key points detected in the first video frame, and the method comprises the following steps of:

[0032] The target object key points corresponding to the first video frame are obtained, and a bounding box corresponding to the target object key points of the first video frame is determined;

[0033] determine a target object region in the second video frame according to a region of the bounding box in the first video frame.

[0034] According to a second aspect of the embodiments of the present disclosure, a device for detecting target object key points is provided, comprising:

[0035] a similarity obtaining unit configured to obtain a similarity between adjacent first and second video frames; the first video frame is in front of the second video frame in terms of video frame sequence, and the second video frame is a current video frame to be detected;

[0036] a first target object identifying unit configured to, when the similarity is less than a preset threshold, invoke an object detection model to determine a target object region in the second video frame;

[0037] a second target object identifying unit configured to, when the similarity is greater than or equal to the preset threshold, determine a target object region in the second video frame based on target object key points that have been detected in the first video frame;

[0038] a key point detecting unit configured to detect target object key points in the second video frame based on the target object region determined from the second video frame.

[0039] In an exemplary embodiment, the key point detecting unit comprises:

[0040] an image input module configured to input an image corresponding to the target object region into a key point detection model, to detect a probability of each pixel point in the image being a target object key point by the key point detection model, and to determine target object key points in the image based on the probability.

[0041] In an exemplary embodiment, the key point detection model is obtained in the following manner:

[0042] a training image obtaining unit configured to obtain a training image containing a target object and real key points corresponding to the target object in the training image; the real key points comprise first type key points and second type key points; the first type key points comprise key points of limb parts, and the second type key points comprise key points of trunk parts;

[0043] a training image input unit configured to input the training image into a neural network model to be trained, to identify a probability of each pixel point in the training image being a target object key point by the neural network model, and to determine a plurality of predicted key points based on the probability.

[0044] The loss function determination unit is configured to determine a loss function of the neural network model by a preset supervision module in the neural network model; wherein the supervision module determines the loss function based on a first loss function corresponding to the real key points and the predicted key points, and a second loss function corresponding to a reference vector and a current vector, the reference vector is a vector representing the relative position between the first type of key points and the second type of key points in the real key points, and the current vector represents the relative position between the predicted first type of key points and the predicted second type of key points in the plurality of predicted key points.

[0045] The neural network model acquisition unit is configured to adjust the parameters of the neural network model based on the loss function until a training end condition is met, and obtain a trained neural network model.

[0046] The supervision module deletion unit is configured to delete the supervision module from the trained neural network model to obtain the key point detection model.

[0047] In an exemplary embodiment, the neural network model comprises a plurality of network modules connected in series, and the network modules are used to output corresponding heat maps for input images; the response value corresponding to each pixel point in the heat map represents the probability that the pixel point is a key point of a target object;

[0048] The training image input unit comprises:

[0049] The image prediction module is configured to take the feature map corresponding to the training image as the input image of the first network module in the neural network model, and take the heat map output by the previous network module as the input image of the subsequent network module.

[0050] The key point probability determination module is configured to determine the probability that each pixel point in the training image is a key point of the target object based on the heat map output by the last network module.

[0051] In an exemplary embodiment, each network module is provided with a corresponding supervision module.

[0052] The loss function determination unit comprises:

[0053] The local loss function determination module is configured to determine, for each network module, a current vector corresponding to a plurality of predicted key points in the heat map output by the network module, and determine a local loss function corresponding to the network module based on a reference vector and a current vector corresponding to the network module by the supervision module corresponding to the network module.

[0054] The loss function calculation module is configured to determine the loss function of the neural network model based on the local loss functions corresponding to the plurality of network modules respectively.

[0055] In an example embodiment, the similarity obtaining unit comprises:

[0056] The first distribution information obtaining module is configured to obtain a first grayscale image corresponding to the first video frame, and determine first distribution information corresponding to pixel values in the first video frame based on the first grayscale image.

[0057] The second distribution information obtaining module is configured to obtain a second grayscale image corresponding to the second video frame, and determine second distribution information corresponding to pixel values in the second video frame based on the second grayscale image.

[0058] The distribution information comparison module is configured to determine the similarity between the first video frame and the second video frame according to the similarity of the first distribution information and the second distribution information.

[0059] In an example embodiment, the second target object recognition unit comprises:

[0060] The bounding box determination module is configured to obtain target object keypoints corresponding to the first video frame, and determine a bounding box corresponding to the target object keypoints in the first video frame.

[0061] The target object region determination module is configured to determine a target object region in the second video frame according to a region of the bounding box in the first video frame.

[0062] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, comprising:

[0063] a processor;

[0064] a memory for storing instructions executable by the processor;

[0065] The processor is configured to execute the instructions to implement the target object keypoint detection method according to any one of the preceding embodiments.

[0066] According to a fourth aspect of embodiments of the present disclosure, a computer readable storage medium is provided, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device can execute the target object keypoint detection method according to any one of the preceding embodiments.

[0067] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises instructions executed by a processor of an electronic device, so that the electronic device can perform the target object key point detection method according to any one of the above aspects.

[0068] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects:

[0069] In the scheme of the present disclosure, the target object region is determined by the object detection model only when the similarity is less than the preset threshold, avoiding frequent calling of the object detection model, saving a large amount of computing resources, and reducing terminal power consumption. In addition, the target object region is determined based on the key points of the previous video frame only when the similarity exceeds the preset threshold, avoiding the incorrect use of the key points of the previous frame, effectively improving the recognition accuracy, and achieving a balance between terminal power consumption and recognition accuracy.

[0070] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0071] The accompanying drawings incorporated in the specification and forming a part of it illustrate the embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0072] Figure 1 is a flowchart of a target object key point detection method according to an exemplary embodiment.

[0073] Figure 2 is a flowchart of a key point detection model training method according to an exemplary embodiment.

[0074] Figure 3 is a structural schematic diagram of a key point detection model according to an exemplary embodiment.

[0075] Figure 4 is a block diagram of a target object key point detection device according to an exemplary embodiment.

[0076] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment.

[0077] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] In order to make ordinary people in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0079] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0080] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0081] Figure 1 This is a flowchart illustrating a method for detecting key points of a target object according to an exemplary embodiment, such as... Figure 1 As shown, this method is illustrated using an example applied to a terminal. It can be understood that this method can also be applied to a server, and to systems where the terminal interacts with the server. Specifically, it may include the following steps.

[0082] In step S110, the similarity between adjacent first and second video frames is obtained.

[0083] The first video frame and the second video frame can be video frames from the video to be processed. The first video frame's frame order precedes the second video frame's frame order, and the second video frame is the video frame currently being detected. The video frame order can be the playback order of multiple video frames, or it can be a pre-set detection order.

[0084] In practical applications, when detecting key points of a target object in a video, multiple video frames in the video can be detected sequentially. Specifically, after detecting the first video frame, the adjacent second video frame to be detected can be obtained, and the similarity between the first and second video frames can be determined.

[0085] In step S120, when the similarity is less than a preset threshold, the object detection model is invoked to determine the target object region in the second video frame;

[0086] As an example, the target object can be a person or other objects whose key points are to be detected, such as animals, buildings, machinery, or landscapes. The target object region can be the image region of the target object in the second video frame.

[0087] An object detection model can be a model that analyzes the image data corresponding to the input image to determine the region where the target object is located in the image. An object detection model can independently identify the target object in the image based on the input image.

[0088] After obtaining the similarity between the first video frame and the second video frame, the similarity can be compared with a preset threshold to determine whether the object detection model needs to be called for the second video frame to be detected.

[0089] In related technologies, taking a human as the target object, frame-by-frame detection of key points on the human body can be performed when detecting these points. Specifically, during frame-by-frame detection, a human body detector is used to detect the position of the human body in each video frame, thereby accurately identifying the human body's position in the current frame when the shooting scene changes. However, while this method can accurately identify the position of the human body in each video frame, it places high demands on the computing power of the terminal device. When users edit videos on low-performance terminal devices, they are prone to experiencing stuttering or even being unable to process the video properly.

[0090] In the embodiments disclosed herein, the similarity between the first video frame and the second video frame can be obtained. When the similarity is less than a preset threshold, it can be determined that the images appearing in the first video frame and the second video frame are significantly different and have low similarity. Therefore, the images corresponding to the first video frame and the second video frame may have scene changes. Thus, at this time, the object detection model can be called to determine the target object region in the second video frame.

[0091] In one example, those skilled in the art can choose the object detection model as needed. For instance, when determining the target object using an anchor-based approach, the single-stage SSD (Single Shot MultiBox Detector) algorithm, which is fast and easy to deploy on mobile devices, can be used, or the two-stage Faster R-CNN algorithm can be used for detection. Of course, the target object can also be determined in an anchor-free manner, such as the CenterNet algorithm.

[0092] In step S130, when the similarity is greater than or equal to the preset threshold, the target object region in the second video frame is determined based on the key points of the target object detected in the first video frame.

[0093] In the related art, the video frames can also be detected according to a pre-interval extraction, that is, the video frames are extracted at a fixed interval, for example, every 5 frames or more than 5 frames. After the video frames are extracted, the target object key points in the current extracted video frame can be determined by continuing to use the target object key points in the previous video frame. Although this method does not need to identify the human body position in each video frame, it can relieve the data processing pressure of the terminal device to a certain extent and reduce the requirement for the computing power of the terminal device. However, when the scene between the video frames changes suddenly, the object detection model cannot be triggered in time, and the target object key points in the previous video frame are continued to be used for detection, resulting in inaccurate key point recognition. Or, when the positions of the target objects in multiple video frames are relatively stable, the object detection model is triggered by mistake, the device power consumption is increased, and resources are wasted.

[0094] In the present disclosure, the similarity between the first video frame and the second video frame can be compared with a preset threshold. When the similarity is less than the threshold, the object detection model is called in time to determine the target object region in the second video frame, so as to avoid continuing to use the information in the first video frame. When the similarity is greater than the threshold, the object detection model does not need to be triggered, and the target object region in the second video frame is determined based on the detected target object key points corresponding to the first video frame.

[0095] Specifically, the first video frame can be a video frame in which the target object key points have been detected. After detecting the next video frame of the first video frame and obtaining the similarity between the first video frame and the second video frame, if it is determined that the similarity is greater than or equal to a preset threshold, it can be determined that the scene or image content of the first video frame and the second video frame is relatively high. Therefore, the object detection model can not be called, and the target object region in the second video frame can be determined based on the detected target object key points of the first video frame.

[0096] In step S140, the target object key points in the second video frame are detected based on the target object region determined from the second video frame.

[0097] As an example, the target object key points can be points used to identify specific parts of a target object in video processing or image processing. For example, when the target object is a human body, the target object key points can be points used to identify different parts of the human body.

[0098] After determining the target object region in the second video frame, the target object key points in the second video frame can be detected based on the target object region determined in the second video frame. Specifically, the image corresponding to the target object region in the second video frame can be identified, for example, a key point recognition algorithm or model is called to determine the target object key points in the image corresponding to the target object region.

[0099] In the method for detecting the target object key point, similarity between adjacent first and second video frames can be obtained, when the similarity is less than a preset threshold, a target object region in the second video frame is determined by calling an object detection model, and when the similarity is greater than or equal to the preset threshold, the target object region in the second video frame is determined based on the target object key point detected in the first video frame. Then, the target object key point in the second video frame can be detected based on the target object region determined in the second video frame. In the scheme of the disclosure, the target object region is determined by the object detection model only when the similarity is less than the preset threshold, which avoids frequent calling of the object detection model, saves a large amount of computing resources, reduces terminal power consumption, and based on the key point of the previous video frame to determine the target object region only when the similarity exceeds the preset threshold, which avoids the error use of the key point of the previous frame, effectively improves the recognition accuracy, and achieves the balance between terminal power consumption and key point recognition accuracy.

[0100] In an example embodiment, whether the similarity is greater than the preset threshold can be used as the only condition for calling the object detection model. By using it as the only condition, the algorithm structure complexity can be greatly reduced, which is beneficial to reduce the power consumption of the terminal device and improve the operation speed of the mobile terminal for video editing.

[0101] In an example embodiment, in step S140, detecting the target object key point in the second video frame based on the target object region determined in the second video frame can include the following steps:

[0102] The image corresponding to the target object region is input into a key point detection model to detect the probability that each pixel point in the image is a target object key point by the key point detection model, and the target object key point in the image is determined based on the probability.

[0103] As an example, the key point detection model can be a model for identifying target object key points in an image.

[0104] In actual application, after determining the target object region in the second video frame, the image corresponding to the target object region in the second video frame can be input into a pre-trained key point detection model, and then the key point detection model can be used to detect the key points of the image, determine the probability that each pixel point in the image is a target object key point, and determine the target object key point in the image based on the probability.

[0105] Specifically, different types of target object key points can exist on the same target object. Taking a human body as an example, the human body can include key points corresponding to multiple body parts such as shoulders, hands, legs, and the like. For each type of target object key point, after obtaining the probability that each pixel point is a key point of the target object, one or more pixel points with a probability higher than a preset probability threshold can be determined as a target object key point.

[0106] In the present disclosure, the probability that each pixel point in an image is a target object key point can be determined by a key point detection model, and the target object key point can be identified based on the probability, thereby achieving accurate identification of the target object key point in the image.

[0107] In an example embodiment, as shown in Figure 2 The key point detection model can be obtained based on the following manner.

[0108] In step S210, a training image containing a target object is obtained, and a real key point corresponding to the target object in the training image is determined.

[0109] As an example, the real key point can be a target object key point pre-labeled in the training image. The real key point can include a first type of key point and a second type of key point. The first type of key point can include a key point of a limb part, and the limb part can include at least one of a left upper limb, a right upper limb, a left lower limb, and a right lower limb. For example, the first type of key point can be a key point on the following parts: a left wrist, a left elbow, a left shoulder, a right wrist, a right elbow, a right shoulder, a left knee, a left ankle, a right knee, a right ankle, and the like. The second type of key point can include a key point of a torso part, such as a key point on a left hip or a right hip.

[0110] In a specific implementation, a training image containing a target object can be obtained, and a real key point corresponding to the target object in the training image can be determined. Specifically, when using a top-down approach for image recognition, i.e., using a single target object as model input, the training image can be an image containing a single target object. A user can pre-label a target object key point corresponding to the target object in the training image to obtain a real key point corresponding to the training image.

[0111] In step S220, the training image is input to a neural network model to be trained to identify the probability that each pixel point in the training image is a key point of the target object by the neural network model, and determine a plurality of predicted key points based on the probability.

[0112] After obtaining the training image, the training image can be input to the neural network model to be trained. After identifying the probabilities of each pixel point in the training image as a key point of the target object by the neural network model, a plurality of predicted key points can be determined from the training image based on the probabilities corresponding to each pixel point, for example, the key points with probabilities higher than a preset probability threshold can be determined as the predicted key points.

[0113] In step S230, the loss function of the neural network model is determined by a preset supervision module in the neural network model.

[0114] The supervision module determines the first loss function corresponding to the real key points and the predicted key points, and determines the second loss function corresponding to the loss function based on the reference vector and the current vector. The reference vector can be a vector representing the relative position between the first type of key points and the second type of key points in the real key points, and the current vector can be a vector representing the relative position between the predicted first type of key points and the predicted second type of key points in the plurality of predicted key points.

[0115] In actual application, the neural network model can be pre-set with a supervision module. After obtaining the plurality of predicted key points in the training image by the neural network model, the supervision module can obtain the first loss function corresponding to the real key points and the predicted key points. At the same time, the supervision module can obtain the reference vector corresponding to the first type of key points and the second type of key points in the real key points, and obtain the current vector corresponding to the predicted first type of key points and the predicted second type of key points in the predicted key points. Thus, the corresponding second loss function can be determined based on the reference vector and the current vector, and the current loss function of the neural network model can be determined based on the first loss function and the second loss function.

[0116] Specifically, the limb movement has flexibility and variability, so that the key points on the limbs also have various movement conditions. In related technologies, when training the key point detection model, it is easy to have the condition that the limbs of the target object have high degrees of freedom and it is difficult to accurately identify the key points on the limbs, the model prediction result deviates greatly from the actual situation, and the learning complexity is increased.

[0117] In the present disclosure, since the key points of the limb parts and the trunk part of the target object have local rigid body properties, the link between the target object and the limbs can be defined and limited by obtaining the vectors corresponding to the first type of key points and the second type of key points. Specifically, after obtaining the first type of key points and the second type of key points in the real key points, the reference vectors corresponding to the first type of key points and the second type of key points can be determined. The reference vectors can be understood as the relative positions between different key points. For example, at least one type of reference vector can be determined in advance, including: left wrist-left elbow, left elbow-left shoulder, left shoulder-left hip, left hip-left knee, left knee-left ankle, right wrist-right elbow, right elbow-right shoulder, right shoulder-right hip, right hip-right knee, right knee-right ankle, left shoulder-right shoulder, and left hip-right hip. By obtaining the reference vectors, the relative positional relationship between the corresponding key points of the limbs or the trunk of the target object during the movement process can be determined.

[0118] After obtaining the plurality of predicted key points, the current vectors corresponding to the predicted first type of key points and the predicted second type of key points in the predicted key points can be determined. The type of the current vectors can be the same as the type of the reference vectors. For example, at least one type of current vector can be included, including: left wrist-left elbow, left elbow-left shoulder, left shoulder-left hip, left hip-left knee, left knee-left ankle, right wrist-right elbow, right elbow-right shoulder, right shoulder-right hip, right hip-right knee, right knee-right ankle, left shoulder-right shoulder, and left hip-right hip.

[0119] After obtaining the reference vectors and the current vectors, the loss function can be determined based on the reference vectors and the current vectors by the supervision module in the neural network model.

[0120] In step S240, the parameters of the neural network model are adjusted based on the loss function until the training end condition is met, and a trained neural network model is obtained.

[0121] In a specific implementation, after obtaining the loss function, the parameters of the neural network model can be adjusted based on the loss function. After adjustment, the neural network model can be repeatedly trained until the training end condition is met, for example, the current iteration number reaches the preset training number or the value of the loss function is lower than the preset training threshold. Then the training can be stopped, and a trained neural network model is obtained.

[0122] In step S250, the supervision module is deleted from the trained neural network model, and the key point detection model is obtained.

[0123] After obtaining the trained neural network model, the supervision module in the model can be deleted to obtain a key point detection model for identifying key points of a target object. By deleting the supervision module from the key point detection model, the normal operation of a terminal device can be ensured after the key point detection model is deployed to the terminal device, such as a mobile terminal, without increasing additional computational complexity.

[0124] In an embodiment of the present disclosure, by adding a supervision module in the training process of the key point detection model, the model training can focus on the link between the limbs and the torso, and the position prediction of the key points of the target object can be limited to a reasonable range, thereby significantly improving the detection accuracy of the key point detection model for the key points of the limbs and the torso of the target object while reducing the complexity of model training.

[0125] In an example embodiment, the neural network model can include a plurality of network modules connected in series, which can be used to output a corresponding heat map for an input image, where the heat map can also be referred to as a Gaussian heat map, and the response value of each pixel point in the heat map represents the probability that the pixel point is a key point of a target object.

[0126] In step S220, inputting the training image into the neural network model to be trained to identify the probability that each pixel point in the training image is a key point of the target object by the neural network model can include:

[0127] The feature map corresponding to the training image is used as the input image of the first network module in the neural network model, and the heat map output by the previous network module is used as the input image of the next network module, and the probability that each pixel point in the training image is a key point of the target object is determined based on the heat map output by the last network module.

[0128] In actual application, the feature map corresponding to the training image can be used as the input image of the first network module in the neural network model, the first network module can generate a corresponding heat map based on the input image, and the heat map can be input into the next network module corresponding to the first network module, and the process can be repeated, so that the heat map output by the previous network module in the plurality of network modules can be used as the input image of the next network module, and a corresponding heat map can be output.

[0129] Until the last network module in the plurality of network modules outputs a corresponding heat map based on the input image of the previous network module corresponding thereto, the probability that each pixel point in the training image is a key point of the target object can be determined based on the response value of each pixel point in the heat map.

[0130] In this disclosure, multiple network modules can be connected in series and the heatmap can be optimized multiple times through multiple network modules. Then, based on the final output heatmap, the probability of each pixel in the training image being a key point of the target object can be determined, which can effectively improve the prediction accuracy of the key points of the target object.

[0131] In an exemplary embodiment, each network module may be provided with a corresponding supervision module. The step of determining the current loss function of the neural network model based on the reference vector and the current vector through the preset supervision module may include:

[0132] For each network module, the current vector corresponding to multiple predicted key points in the heatmap output by the network module is determined, and the local loss function corresponding to the network module is determined by the supervision module corresponding to the network module based on the reference vector and the current vector corresponding to the network module; the loss function of the neural network model is determined based on the local loss functions corresponding to the multiple network modules.

[0133] In practical implementation, a neural network model can have multiple interconnected network modules, and a separate supervision module can be set for each network module.

[0134] When a neural network model acquires predicted keypoints in a training image, for each network module, when the module generates a corresponding heatmap based on the input image, multiple predicted keypoints corresponding to the current heatmap can be determined based on the response values ​​of each pixel in the heatmap. Furthermore, based on the relative positions of these multiple predicted keypoints, the current vectors corresponding to these predicted keypoints can be determined. Then, the local loss function of the corresponding network module can be determined by its corresponding supervision module, based on the reference vector and the current vector.

[0135] After determining the local loss functions corresponding to each of the multiple network modules, the loss function of the neural network model can be determined based on the local loss functions corresponding to each of the multiple network modules.

[0136] like Figure 3 The diagram shown is a schematic representation of the structure of a neural network model in an exemplary embodiment of this disclosure. The neural network model may include a backbone network and multiple network modules following the backbone network.

[0137] The module 301, the module 302 and the residual module can be included in the backbone network, and six network modules in series can be included after the backbone network, and one module 301 and one module 302 can be included in each network module. When training the neural network model, a training image can be input into the neural network model. After receiving the training image, the neural network model can obtain an original feature map corresponding to the training image through the module 301, and perform linear operation on the original feature map to obtain a plurality of simulated feature maps corresponding to the original feature map. Then, the original feature map and the simulated feature maps can be spliced and input into the module 302. In an example, the feature map generated by the module 301 can be a heat map, and the module 301 can be a Ghostnet module. The Ghostnet module generates a feature map by using fewer convolution kernels, and further efficiently generates more simulated feature maps through a simple linear transformation operation, and finally connects them and outputs them, so that a plurality of feature maps can be generated on a mobile terminal in a simple and easy way.

[0138] After obtaining the spliced feature map, the module 302 can extract features of different levels of the spliced feature map to obtain a high-resolution feature map and a low-resolution feature map, and perform a fusion operation on the high-resolution feature map and the low-resolution feature map. Then, the fused feature map can be output to the residual module. The feature map generated by the module 302 can be a heat map, the module 302 can be an Hourglass module, and the residual module can be a Dense module. In this example, by stacking the module 301, the module 302 and the residual module, more accurate target object key points can be detected under the same amount of calculation.

[0139] After the residual module extracts features of the input feature map and outputs to the network module 1, the network module 1 can generate a corresponding heat map through the module 301 and the module 302 therein. The supervision module corresponding to the network module 1 can determine a current vector based on the heat map output by the network module 1. Then, the local loss function-1 corresponding to the network module 1 can be determined according to the current vector corresponding to the network module 1 and the reference vector determined in advance. Correspondingly, the subsequent multiple network modules in series can determine the local loss function-2, …, the local loss function-6 in the same way. Finally, the loss function corresponding to the neural network model can be determined based on the local loss function-1 to the local loss function-6, and the parameters of the neural network model can be adjusted based on the loss function.

[0140] In the present disclosure, by determining the loss function of the neural network model based on the local loss functions corresponding to the plurality of network modules, the overall parameters of the model can be more accurately optimized based on the local loss functions corresponding to the plurality of network modules on the basis of the optimized prediction of the heat map by the plurality of network modules, which is conducive to obtaining more accurate target object key point recognition results.

[0141] In an example embodiment, the key point detection model can also make full use of the temporal information between a plurality of video frames for key point detection, for example, using a DCPose module in the key point detection model to increase the influence information of the previous and subsequent video frames on the current video frame, or using an Hrnet structure, etc.

[0142] In an example embodiment, in step S110, the similarity between the adjacent first video frame and the second video frame can include the following steps:

[0143] The first gray scale image corresponding to the first video frame is obtained, and the first distribution information corresponding to the pixel value in the first video frame is determined based on the first gray scale image. The second gray scale image corresponding to the second video frame is obtained, and the second distribution information corresponding to the pixel value in the second video frame is determined based on the second gray scale image. The similarity between the first video frame and the second video frame is determined according to the similarity of the first distribution information and the second distribution information.

[0144] As an example, the pixel value in the first video frame can be the pixel value corresponding to each pixel point in the first gray scale image; the pixel value in the second video frame can be the pixel value corresponding to each pixel point in the second gray scale image.

[0145] The first distribution information can be information representing the distribution rule of the pixel value in the first gray scale image, and the second distribution information can be information representing the distribution rule of the pixel value in the second gray scale image. The first distribution information and the second distribution information can be histograms representing the distribution of the pixel value.

[0146] In a specific implementation, the similarity can be determined based on the histogram corresponding to the video frame, which has the characteristics of fast operation speed and accurate similarity calculation. Specifically, after obtaining the second video frame to be detected, the first video frame adjacent to the second video frame can be determined, and the first gray scale image corresponding to the first video frame can be obtained. After obtaining the first gray scale image, the pixel value corresponding to each pixel point in the first gray scale image can be obtained, and the distribution of a plurality of pixel values can be counted, for example, a histogram representing the distribution of the pixel value is obtained, and the first distribution information corresponding to the pixel value in the first video frame is obtained.

[0147] Similarly, for the second video frame, after obtaining the gray image corresponding to the second video frame, the pixel values corresponding to the pixel points in the second gray image can be obtained, and the distribution statistics of the plurality of pixel values are performed to obtain the second distribution information corresponding to the pixel values in the second video frame.

[0148] After obtaining the first distribution information and the second distribution information, the similarity of the first distribution information and the second distribution information can be obtained, for example, when the first distribution information and the second distribution information are histograms, the similarity of the first distribution information and the second distribution information can be determined by comparing the fitting degree of the two histograms.

[0149] After determining the similarity of the first distribution information and the second distribution information, the similarity between the first video frame and the second video frame can be determined based on the similarity.

[0150] In the present disclosure, by determining the first distribution information corresponding to the pixel values in the first video frame and the second distribution information corresponding to the pixel values in the second video frame, the similarity between the first video frame and the second video frame can be determined according to the similarity of the first distribution information and the second distribution information, which realizes the rapid and accurate determination of the similarity between the video frames by comparing the distribution information of the pixel values in the image, and provides an accurate basis for judgment while avoiding increasing the power consumption of the terminal device.

[0151] In another example, the similarity calculation algorithm can also be determined according to the actual situation, for example, the Cosine similarity algorithm, the pHash (Perceptual hash algorithm) or the content feature algorithm can also be used to calculate the similarity between the first video frame and the second video frame.

[0152] In an exemplary embodiment, the target object region in the second video frame is determined based on the target object key points detected in the first video frame, including:

[0153] The target object key points corresponding to the first video frame are obtained, and the bounding box corresponding to the target object key points of the first video frame is determined; and the target object region in the second video frame is determined according to the region of the bounding box in the first video frame.

[0154] In a specific implementation, when the similarity between the first video frame and the second video frame is greater than or equal to a preset threshold, the target object key points detected in the first video frame can be obtained, and the bounding box corresponding to the target object key points can be determined.

[0155] Specifically, in identifying the key points of the target object, two ways can be used, i.e., top-down or bottom-up, or taking the whole image or a single target object as input to identify the key points. In this embodiment, when taking a single target object as input, for each target object in the first video frame, the key points of the target object corresponding to the target object can be obtained, and the bounding box corresponding to the key points of the target object is determined, for example, the minimum bounding rectangle corresponding to the key points is obtained.

[0156] After obtaining the bounding box corresponding to the key points of the target object in the first video frame, the target object region in the second video frame can be determined according to the region of the bounding box in the first video frame, for example, the region of the bounding box in the first video frame can be obtained, and the coordinates corresponding to the region are determined as the coordinates corresponding to the target object region in the second video frame, thereby locating the target object region in the second video frame. Of course, on the basis of the coordinates corresponding to the bounding box, corresponding offset processing can be performed, such as adding a corresponding offset, and then the target object region in the second video frame can be determined based on the processed coordinates.

[0157] In related technologies, although the key points of the target object can be determined in the region of the target object in the image after identifying the region of the target object in the image by frame-by-frame identification of multiple video frames, when multiple video frames are detected independently, and when the detection results of the target object region between video frames are greatly different, the key points of the target object between video frames are likely to be obviously jittered, which brings many inconveniences to subsequent video editing, such as jitter of the beautification effect of the target object, unnatural transition between frames, and the like.

[0158] In this disclosure, when the similarity between the first video frame and the second video frame is greater than or equal to a preset threshold, the region corresponding to the target object in the second video frame is determined based on the region where the key points of the target object in the first video frame are located, i.e., the region where the key points of the target object in the first video frame are located, which can avoid obvious differences in the positions of the target object between the first video frame and the second video frame when the video scene is similar, for example, when no scene mutation occurs, reduce the jitter of the key points of the target object between video frames, make the transition of the key points of the target object between video frames natural, and optimize the beautification effect of the video.

[0159] It should be understood that, although Figure 1 , Figure 2 The steps in the flowchart of Figure 1 , Figure 2At least one of the steps in the above method can include multiple steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the steps or stages is not necessarily sequential, but can be performed alternately or alternately with at least one of the other steps or steps in the other steps.

[0160] It can be understood that the same / similar parts between each embodiment of the above method in the specification can be referred to each other, and each embodiment focuses on the difference from other embodiments, and the related part can be referred to the description of other method embodiments.

[0161] Figure 4 is a detection device block diagram of a target object key point according to an exemplary embodiment. Referring to Figure 4 The device includes a similarity obtaining unit 401, a first target object identification unit 402, a second target object identification unit 403, and a key point detection unit 404.

[0162] The similarity obtaining unit 401 is configured to perform obtaining the similarity between the adjacent first video frame and the second video frame; the video frame sequence of the first video frame is before the second video frame, and the second video frame is the current video frame to be detected;

[0163] The first target object identification unit 402 is configured to perform when the similarity is less than a preset threshold, calling an object detection model to determine the target object region in the second video frame;

[0164] The second target object identification unit 403 is configured to perform when the similarity is greater than or equal to the preset threshold, determining the target object region in the second video frame based on the target object key point detected in the first video frame;

[0165] The key point detection unit 404 is configured to perform detecting the target object key point in the second video frame based on the target object region determined from the second video frame.

[0166] In an exemplary embodiment, the key point detection unit includes:

[0167] The image input module is configured to perform inputting the image corresponding to the target object region into a key point detection model, to detect the probability of each pixel point in the image being a target object key point through the key point detection model, and to determine the target object key point in the image based on the probability.

[0168] In an exemplary embodiment, the key point detection model is obtained based on the following way:

[0169] The training image acquisition unit is configured to acquire a training image containing a target object and real key points corresponding to the target object in the training image; the real key points include first type key points and second type key points; the first type key points include key points of limb parts, and the second type key points include key points of trunk parts;

[0170] The training image input unit is configured to input the training image into a neural network model to be trained, to identify, by the neural network model, a probability that each pixel point in the training image is a key point of the target object, and to determine a plurality of predicted key points based on the probability.

[0171] The loss function determination unit is configured to determine, by a preset supervision module in the neural network model, a current loss function of the neural network model; wherein the supervision module determines the loss function based on a first loss function corresponding to the real key points and the predicted key points, and a second loss function corresponding to a reference vector and a current vector; the reference vector is a vector representing a relative position between the first type key points and the second type key points in the real key points, and the current vector is a vector representing a relative position between a predicted first type key point and a predicted second type key point in the plurality of predicted key points.

[0172] The neural network model acquisition unit is configured to adjust parameters of the neural network model based on the loss function until a training end condition is met, to obtain a trained neural network model.

[0173] The supervision module deletion unit is configured to delete the supervision module from the trained neural network model to obtain the key point detection model.

[0174] In an exemplary embodiment, the neural network model includes a plurality of network modules connected in series, and the network modules are used to output corresponding heat maps for input images; a response value corresponding to each pixel point in the heat map represents a probability that the pixel point is a key point of a target object.

[0175] The training image input unit includes:

[0176] The image prediction module is configured to take a feature map corresponding to the training image as an input image of a first network module in the neural network model, and take a heat map output by a previous network module as an input image of a subsequent network module.

[0177] The key point probability determination module is configured to determine, based on a heat map output by a last network module, a probability that each pixel point in the training image is a key point of the target object.

[0178] In an example embodiment, each of the network modules is correspondingly provided with a supervision module;

[0179] The loss function determination unit comprises:

[0180] The local loss function determination module is configured to determine, for each of the network modules, a current vector corresponding to a plurality of predicted key points in a heat map output by the network module, and determine, by the supervision module corresponding to the network module, a local loss function corresponding to the network module based on a reference vector and the current vector corresponding to the network module;

[0181] The loss function calculation module is configured to determine the loss function of the neural network model based on the local loss functions respectively corresponding to the plurality of network modules.

[0182] In an example embodiment, the similarity obtaining unit comprises:

[0183] The first distribution information obtaining module is configured to obtain a first grayscale image corresponding to the first video frame, and determine first distribution information corresponding to pixel values in the first video frame based on the first grayscale image;

[0184] The second distribution information obtaining module is configured to obtain a second grayscale image corresponding to the second video frame, and determine second distribution information corresponding to pixel values in the second video frame based on the second grayscale image;

[0185] The distribution information comparison module is configured to determine the similarity between the first video frame and the second video frame according to the similarity of the first distribution information and the second distribution information.

[0186] In an example embodiment, the second target object identification unit comprises:

[0187] The bounding box determination module is configured to obtain target object key points corresponding to the first video frame, and determine a bounding box corresponding to the target object key points of the first video frame;

[0188] The target object region determination module is configured to determine a target object region in the second video frame according to a region of the first video frame in the bounding box.

[0189] As to the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be described in detail here.

[0190] Figure 5is a block diagram of an electronic device 500 for implementing a method of detecting a key point of a target object according to an exemplary embodiment. The electronic device 500 can be, for example, a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0191] Referring to Figure 5 The electronic device 500 can include one or more of the following components: a processing component 502, a memory 504, a power supply component 506, a multimedia component 508, an audio component 510, an input / output (I / O) interface 512, a sensor component 514, and a communication component 516.

[0192] The processing component 502 generally controls the overall operations of the electronic device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 can include one or more processors 520 to execute instructions to complete all or part of steps of the above methods. In addition, the processing component 502 can include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 can include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0193] The memory 504 is configured to store various types of data to support operations of the electronic device 500. Examples of these data include instructions for any application or method operating on the electronic device 500, contact data, phonebook data, messages, pictures, videos, and the like. The memory 504 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene memory.

[0194] The power supply component 506 supplies power for the various components of the electronic device 500. The power supply component 506 can include a power supply management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 500.

[0195] The multimedia component 508 includes a screen to provide an output interface between the electronic device 500 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and intensity of the touching or sliding action. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the electronic device 500 is in an operation mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom ability.

[0196] The audio component 510 is configured to output and / or input an audio signal. For example, the audio component 510 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 500 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 504 or transmitted via the communication component 516. In some embodiments, the audio component 510 further includes a speaker for outputting an audio signal.

[0197] The I / O interface 512 provides an interface between the processing component 502 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0198] The sensor component 514 includes one or more sensors to provide various state assessments for the electronic device 500. For example, the sensor component 514 can detect an open / closed state of the electronic device 500, relative positioning of components, such as a display and a keypad of the electronic device 500, a change in position of the electronic device 500 or components of the electronic device 500, presence or absence of user contact with the electronic device 500, a direction or acceleration / deceleration of the device 500, and a change in temperature of the electronic device 500. The sensor component 514 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 514 can further include a light sensor such as a CMOS or CCD image sensor for use in an imaging application. In some embodiments, the sensor component 514 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0199] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 500 and other devices. The electronic device 500 can access a wireless network based on a communication standard, such as WiFi, a cellular network standard (such as 2G, 3G, 4G, or 5G), or a combination thereof. In an example embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 516 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0200] In an example embodiment, the electronic device 500 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors, or other electronic elements, for performing the above-described methods.

[0201] In an example embodiment, a computer-readable storage medium including instructions, such as the memory 504 including instructions, is also provided, which can be executed by the processor 520 of the electronic device 500 to complete the above-described methods. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0202] In an example embodiment, a computer program product including instructions, which can be executed by the processor 520 of the electronic device 500 to complete the above-described methods, is also provided.

[0203] Figure 6 FIG. 6 is a block diagram of an electronic device 600 for implementing a method of detecting a key point of a target object according to an example embodiment. For example, the electronic device 600 can be a server. Referring to FIG. 6, Figure 6 The electronic device 600 includes a processing component 620, which further includes one or more processors, and a memory resource represented by a memory 622, for storing instructions executable by the processing component 620, such as an application program. The application program stored in the memory 622 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 620 is configured to execute the instructions to perform the above-described methods.

[0204] The electronic device 600 can further include a power supply component 624 configured to perform power management of the electronic device 600, a wired or wireless network interface 626 configured to connect the electronic device 600 to a network, and an input / output (I / O) interface 628. The electronic device 600 can operate based on an operating system stored in the memory 622, such as Windows Server, Mac OS X, Unix, Linux, FreeBDS, or the like.

[0205] In an exemplary embodiment, a computer readable storage medium including instructions, such as the memory 622 including instructions, is also provided, which can be executed by the processor of the electronic device 600 to complete the above method. The storage medium can be a computer readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0206] In an exemplary embodiment, a computer program product including instructions, which can be executed by the processor of the electronic device 600 to complete the above method, is also provided.

[0207] It should be noted that the above-described apparatus, electronic device, computer readable storage medium, computer program product, etc. according to the description of the method embodiments can also include other implementations, and the specific implementation can be referred to the description of the related method embodiments, which will not be described here.

[0208] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations contained within the spirit and scope of the features of the present disclosure. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0209] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings and that various modifications and changes can be made therein without departing from the scope thereof. The scope of the present disclosure is indicated by the appended claims.

Claims

1. A method for detecting a key point of a target object, characterized in that, The method comprises: obtaining a similarity between adjacent first and second video frames; the video frame sequence of the first video frame is before the second video frame, and the second video frame is a current video frame to be detected; when the similarity is less than a preset threshold, inputting the second video frame into an object detection model, and determining a target object region of each target object in the second video frame by the object detection model; when the similarity is greater than or equal to the preset threshold, determining a target object region of each target object in the second video frame based on a bounding box of each target object key point that has been detected in the first video frame; detecting target object key points in the second video frame based on the target object region determined from the second video frame.

2. The method of claim 1, wherein, The method of detecting target object key points in the second video frame based on the target object region determined from the second video frame comprises: inputting an image corresponding to the target object region into a key point detection model to detect a probability that each pixel point in the image is a target object key point by the key point detection model, and determining target object key points in the image based on the probability.

3. The method of claim 2, wherein, The key point detection model is obtained based on the following method: obtaining a training image containing a target object and real key points corresponding to the target object in the training image; the real key points comprise first type key points and second type key points; the first type key points comprise key points of limb parts, and the second type key points comprise key points of trunk parts; inputting the training image into a neural network model to be trained to identify a probability that each pixel point in the training image is a target object key point by the neural network model, and determining a plurality of predicted key points based on the probability; determining a loss function of the neural network model at present by a preset supervision module in the neural network model; wherein the supervision module determines the loss function based on a first loss function corresponding to the real key points and the predicted key points, and a second loss function corresponding to a reference vector and a current vector; the reference vector is a vector representing relative positions between the first type key points and the second type key points in the real key points, and the current vector represents relative positions between predicted first type key points and predicted second type key points in the plurality of predicted key points; adjusting parameters of the neural network model based on the loss function until a training end condition is met, to obtain a trained neural network model; deleting the supervision module from the trained neural network model to obtain the key point detection model.

4. The method of claim 3, wherein, The neural network model comprises a plurality of network modules connected in series, and each network module is configured to output a corresponding heat map for an input image; a response value corresponding to each pixel point in the heat map represents a probability that the pixel point is a target object key point. The method of inputting the training image into the neural network model to be trained to identify a probability that each pixel point in the training image is a target object key point by the neural network model comprises: The feature map corresponding to the training image is taken as an input image of a first network module in the neural network model, and a heat map output by a previous network module is taken as an input image of a subsequent network module; Based on the heat map output by the last network module, a probability of each pixel point in the training image being a key point of the target object is determined.

5. The method of claim 4, wherein, Each network module is provided with a supervision module; The current loss function of the neural network model is determined based on the reference vector and the current vector by the preset supervision module, and the current loss function of the neural network model comprises: For each network module, a current vector corresponding to a plurality of predicted key points in a heat map output by the network module is determined, and a local loss function corresponding to the network module is determined based on the reference vector and the current vector corresponding to the network module by the supervision module corresponding to the network module. The loss function of the neural network model is determined based on the local loss functions corresponding to the plurality of network modules.

6. The method of claim 1, wherein, The similarity between the adjacent first video frame and the second video frame is obtained, comprising: A first gray image corresponding to the first video frame is obtained, and first distribution information corresponding to pixel values in the first video frame is determined based on the first gray image; A second gray image corresponding to the second video frame is obtained, and second distribution information corresponding to pixel values in the second video frame is determined based on the second gray image; The similarity between the first video frame and the second video frame is determined according to the similarity of the first distribution information and the second distribution information.

7. The method of claim 1, wherein, The target object region of each target object in the second video frame is determined based on the bounding box of the detected target object key point of each target object in the first video frame, comprising: The target object key point corresponding to each target object in the first video frame is obtained, and the bounding box corresponding to the target object key point of each target object in the first video frame is determined; The target object region of each target object in the second video frame is determined according to the region of the bounding box of each target object in the first video frame.

8. An apparatus for detecting a key point of a target object, characterized by comprising: Comprising: The similarity acquisition unit is configured to obtain the similarity between the adjacent first video frame and the second video frame; the video frame order of the first video frame is before the second video frame, and the second video frame is a current video frame to be detected; The first target object identification unit is configured to input the second video frame into an object detection model when the similarity is less than a preset threshold, and determine the target object region in the second video frame by the object detection model; The second target object identification unit is configured to determine the target object region in the second video frame based on the detected target object key point of the first video frame when the similarity is greater than or equal to the preset threshold; The key point detection unit is configured to detect the target object key point in the second video frame based on the target object region determined from the second video frame.

9. The apparatus of claim 8, wherein, The key point detection unit comprises: The image input module is configured to input the image corresponding to the target object region into a key point detection model, to detect the probability of each pixel point in the image being a target object key point through the key point detection model, and to determine the target object key point in the image based on the probability.

10. The apparatus of claim 9, wherein, The key point detection model is obtained based on the following manner: The training image acquisition unit is configured to acquire a training image containing a target object and real key points corresponding to the target object in the training image; the real key points include first type key points and second type key points; the first type key points include key points of limb parts, and the second type key points include key points of trunk parts; The training image input unit is configured to input the training image into a neural network model to be trained, to identify the probability of each pixel point in the training image being a target object key point through the neural network model, and to determine a plurality of predicted key points based on the probability. The loss function determination unit is configured to determine a current loss function of the neural network model through a preset supervision module in the neural network model; wherein the supervision module determines the loss function based on a first loss function corresponding to the real key points and the predicted key points, and a second loss function corresponding to a reference vector and a current vector; the reference vector is a vector representing the relative position between the first type key points and the second type key points in the real key points, and the current vector is a vector representing the relative position between the predicted first type key points and the predicted second type key points in the plurality of predicted key points. The neural network model acquisition unit is configured to adjust the parameters of the neural network model based on the loss function until a training end condition is met, to obtain a trained neural network model. The supervision module deletion unit is configured to delete the supervision module from the trained neural network model to obtain the key point detection model.

11. The apparatus of claim 10, wherein, The neural network model includes a plurality of network modules connected in series, and each network module is configured to output a corresponding heat map for an input image; the response value corresponding to each pixel point in the heat map represents the probability of the pixel point being a target object key point. The training image input unit includes: The image prediction module is configured to input a feature map corresponding to the training image as an input image of a first network module in the neural network model, and input a heat map output by a previous network module as an input image of a subsequent network module. The key point probability determination module is configured to determine the probability of each pixel point in the training image being a target object key point based on a heat map output by a last network module.

12. The apparatus of claim 11, wherein, Each network module is provided with a supervision module; The loss function determination unit includes: The local loss function determination module is configured to determine, for each network module, a current vector corresponding to the plurality of predicted key points in the heat map output by the network module, and determine, by the supervision module corresponding to the network module, a local loss function corresponding to the network module based on the reference vector and the current vector corresponding to the network module. The loss function calculation module is configured to determine the loss function of the neural network model based on the local loss functions corresponding to the plurality of network modules.

13. The apparatus of claim 8, wherein, The similarity obtaining unit comprises: The first distribution information obtaining module is configured to obtain a first grayscale image corresponding to the first video frame, and determine first distribution information corresponding to pixel values in the first video frame based on the first grayscale image. The second distribution information obtaining module is configured to obtain a second grayscale image corresponding to the second video frame, and determine second distribution information corresponding to pixel values in the second video frame based on the second grayscale image. The distribution information comparison module is configured to determine the similarity between the first video frame and the second video frame according to the similarity of the first distribution information and the second distribution information.

14. The apparatus of claim 8, wherein, The second target object recognition unit comprises: The bounding box determination module is configured to obtain target object key points corresponding to each target object in the first video frame, and determine a bounding box corresponding to the target object key points of each target object in the first video frame. The target object region determination module is configured to determine a target object region of each target object in the second video frame according to the region of the bounding box of each target object in the first video frame.

15. An electronic device, comprising: The processor; Memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the target object key point detection method of any one of claims 1-7. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute the target object key point detection method of any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The instructions are executed by the processor of the electronic device, so that the electronic device can execute the target object key point detection method of any one of claims 1-7.

17. A computer program product, comprising instructions therein, characterised in that, ​

Images