Key point position determination method and device

Abstract

The present application provides a method and device for determining key point positions. For the situation of detecting key point positions in each frame in a video, the method includes: obtaining the predicted position and confidence level of a target key point in a current video frame in a video to be processed; determining the target position of the target key point in the current video frame according to the target position of the target key point in a previous video frame, the predicted position and confidence level of the target key point in the current video frame, processing the video to be processed based on the target position of the target key point, determining the position of the key point in the current frame in combination with the confidence level and the position of the key point in adjacent frames, thereby improving the accuracy of key point position detection, reducing the jitter of the key point in the processed video, and improving the quality of video processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for determining key point positions. Background Art

[0002] In video processing scenarios, such as live broadcasting and short video, in order to perform body beautification, facial enhancement, and pre-effect processing on the portraits in the video, it is necessary to identify the portraits in each frame of the video based on a position detection algorithm.

[0003] Since the position detection algorithm is aimed at a single frame image, in order to reduce the jitter of key points between frames or video frames in the video to be processed, the weighted average of the position coordinates of the key points in the previous and next two or more frames of the image, or the low-pass filtering method, is usually used to determine the position of the key point in the next frame.

[0004] However, when using the above key point position correction method, due to improper determination of the weighting coefficient, the key point jitter problem still exists in some scenes, such as motion scenes. Summary of the invention

[0005] The present application provides a method and device for determining the position of key points, which determines the position coordinates of key points in a subsequent frame based on the position coordinates of key points in adjacent frames and their confidence levels, thereby improving the accuracy of key point determination and effectively reducing the jitter of key points in a video to be processed.

[0006] In a first aspect, the present application provides a method for determining a key point position, the method comprising:

[0007] Obtain the predicted position and confidence of the target key point in the current video frame in the video to be processed; determine the target position of the target key point in the current video frame according to the target position of the target key point in the previous video frame, the predicted position and confidence of the target key point in the current video frame, so as to process the video to be processed based on the target position of the target key point.

[0008] In a second aspect, the present application provides a key point position determination device, the device comprising:

[0009] A predicted position acquisition module is used to obtain the predicted position and confidence of the target key point in the current video frame in the video to be processed; a target position determination module is used to determine the target position of the target key point in the current video frame based on the target position of the target key point in the previous video frame, the predicted position of the target key point in the current video frame and its confidence, so as to process the video to be processed based on the target position of the target key point.

[0010] In a third aspect, the present application provides an electronic device, including:

[0011] A processor, and a memory communicatively connected to the processor;

[0012] The memory stores computer-executable instructions;

[0013] The processor executes the computer-executable instructions stored in the memory to implement the key point position determination method provided in the first aspect of the present application.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the key point position determination method provided in the first aspect of the present application.

[0015] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the key point position determination method provided in the first aspect of the present application.

[0016] The key point position determination method and device provided in the present application are aimed at the application scenarios of video processing. The position of each video frame in the to-be-processed video is detected in advance to obtain the predicted position of the target key point in each video frame and its confidence. For the current video frame, the target position of the target key point in the current video frame is determined according to the target position of the target key point in the previous video frame of the current video frame and the predicted position of the target key point in the current video frame and its confidence, so as to obtain the target position of each target key point in each video frame in the to-be-processed video. Based on the target position of the target key point, the to-be-processed video is processed, such as body beautification processing, face beautification processing, etc., and the position coordinates of the key point in the next frame are determined based on the confidence of the predicted position and the position of the key point in the adjacent frames, thereby improving the accuracy of key point determination, effectively reducing the jitter of the key point in the processed video, improving the video processing quality, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] Figure 1 A schematic diagram of a flow chart of a method for determining a key point position provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of a flow chart of a method for determining a key point position provided in another embodiment of the present application;

[0020] Figure 3A schematic diagram of a result of determining the positions of key points of a human body provided in one embodiment of the present application;

[0021] Figure 4 A schematic diagram of a flow chart of a method for determining a key point position provided in another embodiment of the present application;

[0022] Figure 5 A schematic diagram of a flow chart of a method for determining a key point position provided in another embodiment of the present application;

[0023] Figure 6 A schematic diagram of the structure of a key point position determination device provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0025] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] In video processing scenarios, such as body beautification, face beautification, face recognition or object recognition in videos, it is necessary to identify the body, face or other objects, such as cars and animals, in each frame. Usually, the positions of target objects such as the body and face in each frame of the video to be processed are determined by marking the key points.

[0028] Usually, the positions of key points in each frame image in the video to be processed are determined based on key point position detection algorithms, such as point regression algorithms, point classification algorithms, model-based detection algorithms, etc.

[0029] Since the movement of key point positions between frames in the video to be processed is continuous, in order to improve the accuracy of key point position determination and reduce the jitter of key points in the video, the weighted average of the positions of the key points in adjacent frames or multiple consecutive frames is usually used to determine the final position of the key point in the current frame.

[0030] However, due to the different motion amplitudes of the target objects in the video, it is difficult to determine the weighting coefficients of the key point positions in each frame, resulting in poor accuracy in the key point position correction, incomplete jitter removal, or obvious delays in the video. For example, in a scene where an arm slowly moves out of the camera's field of view, since the position deviation of the key points of the arm in adjacent frames is small, if the weighting coefficient is not set properly, it will cause the arm to have jitters, delays, and other incoherent movements in the processed video, and even cause the arm to appear in a video frame where the arm does not exist, resulting in poor video processing results. Moreover, in related technologies, it is often only possible to process video frames where key points are detected, and video frames where key points are not detected are often not processed, resulting in the intermittent presence and absence of key points in the video and incoherent movement of key points.

[0031] In order to improve the solution to the key point jitter problem and improve the accuracy of key point position determination, an embodiment of the present application provides a key point position determination method, which determines the position of the target key point in the current video frame based on the position and confidence of the target key point in the previous frame image and the predicted position and confidence of the target key point detected in the current video frame. By introducing the confidence parameter, the accuracy of key point position determination is improved, and the phenomenon of key point jitter in the video is effectively suppressed.

[0032] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0033] Figure 1 A flowchart of a key point location determination method provided in an embodiment of the present application is provided. The key point location determination method can be executed by any device with image processing function, which can be in the form of a computer, a server, etc. Figure 1 As shown, the key point position determination method includes the following steps:

[0034] Step S101, obtaining the predicted position and confidence level of the target key point in the current video frame in the video to be processed.

[0035] The video to be processed can be a video of any length, such as a short video of 15s or 30s, and the target key point is a feature point of the target object in the video to be processed. The current video frame is a video frame in which the target position of the target key point has not been determined in the video to be processed, and the target position of the target key point in the previous video frame (referred to as the previous video frame) of the current video frame has been determined.

[0036] Exemplarily, a user can upload the video to be processed to a preset application on the user terminal through a user terminal or other terminal with a camera function. The server corresponding to the preset application receives the video to be processed and executes the key point position determination method provided in the present application to obtain the target position of each key point in each video frame or at least part of the video frame in the video to be processed.

[0037] Exemplarily, the target object may be a face, a body, an animal, a plant, a car, a sign, etc.

[0038] Exemplarily, the target object may be a human face or a human body, and the target key point may be a feature point of each part of the human face or human body, such as a feature point on an eye, an arm, a buttock, a leg, or the like.

[0039] Specifically, based on the position detection algorithm, the position detection of key points can be performed on each video frame or at least part of the video frames in the processed video to obtain the detection results of each video frame or each frame in the at least part of the video frames. The detection results include the predicted positions of each target key point in the video frame and their confidence. The position detection algorithm can be any algorithm that outputs the predicted position and its confidence, such as a position detection algorithm based on deep learning, such as a position detection algorithm based on a heat map (Heatmap), etc. The confidence of the predicted position is used to identify the probability that the target key point is located at the predicted position, and the confidence can be expressed in the form of percentage, decimal, positive integer, etc.

[0040] Step S102, determining the target position of the target key point in the current video frame according to the target position of the target key point in the previous video frame, the predicted position of the target key point in the current video frame and its confidence, so as to process the video to be processed based on the target position of the target key point.

[0041] The target position and the predicted position can both be represented by position coordinates, such as (x, y), where x and y represent the row and column of the pixel point corresponding to the target key point, respectively.

[0042] In one embodiment, the current video frame may also be referred to as a current frame image, and the previous video frame may also be referred to as a previous frame image.

[0043] Specifically, if target key points are detected in both the current video frame and its previous video frame, the target position of the target key point in the current video frame can be determined based on the target position of the target key point in the previous video frame, the predicted position of the target key point in the current video frame and its confidence level, and so on, to obtain the target position of each target key point in each video frame or at least part of the video frames in the video to be processed.

[0044] For the first video frame of the video to be processed, if the detection result of the first video frame includes the predicted position of the target key point and its confidence, the predicted position is directly determined as the target position of the target key point. Correspondingly, the confidence of the predicted position is the confidence of the target position, so as to determine the target position of the target key point in the second video frame based on the target position of the target key point in the first video frame and the predicted position of the target key point in the second video frame and its confidence. For the Nth (N≥2) video frame in the video to be processed, the target position of the target key point in the N-1th video frame is determined based on the target position of the target key point in the Nth video frame and the predicted position of the target key point in the Nth video frame and its confidence.

[0045] Specifically, for each target key point, the target position of the target key point in the current video frame can be determined based on the position deviation between the predicted position of the target key point in the current video frame (such as the Nth video frame) and the target position of the target key point in the previous video frame (such as the N-1th video frame), and the confidence of the predicted position of the target key point in the current video frame.

[0046] Furthermore, when determining the target position of the target key point in the Nth video frame, i.e., the current video frame, the target position and confidence of the target key point in the previous video frame can be obtained, and then the target position and confidence of the target key point in the current video frame can be determined based on the target position and confidence of the target key point in the previous video frame and the predicted position and confidence of the target key point in the current video frame.

[0047] Specifically, for each target key point, the target position of the target key point in the current video frame can be determined based on the position deviation between the predicted position of the target key point in the current video frame (such as the Nth video frame) and the target position of the target key point in the previous video frame (such as the N-1th video frame), the confidence of the target position of the target key point in the previous video frame, and the confidence of the predicted position of the target key point in the current video frame.

[0048] Specifically, the video to be processed may be processed based on the target position of the target key point in at least part of the video frames of the video to be processed, such as face beautification processing, body beautification processing, etc. The at least part of the video frames may be all the video frames of the video to be processed, or may be a specified one or more frames in the video to be processed, or may be each video frame in which the target key point is detected in the video to be processed.

[0049] The key point position determination method provided in the present embodiment, for the application scenario of video processing, performs position detection on each video frame in the to-be-processed video in advance to obtain the predicted position of the target key point in each video frame and its confidence; for the current video frame, the target position of the target key point in the current video frame is determined according to the target position of the target key point in the previous video frame of the current video frame and the predicted position of the target key point in the current video frame and its confidence, thereby obtaining the target position of each target key point in each video frame in the to-be-processed video; based on the target position of the target key point, the to-be-processed video is processed, such as body beautification processing, face beautification processing, etc., thereby realizing the determination of the position coordinates of the key point in the next frame based on the confidence of the predicted position and the position of the key point in the adjacent frames, thereby improving the accuracy of key point determination, effectively reducing the jitter of the key point in the processed video, improving the video processing quality, and improving the user experience.

[0050] Figure 2 A flowchart of a key point location determination method provided in another embodiment of the present application is provided. This embodiment is directed to the situation where target key points are detected in both the current video frame and the previous video frame. Figure 1 On the basis of the embodiment shown in the figure, step S102 is further refined, and after step S102, a step of determining the confidence of the target position of the target key in the current video frame is added, such as Figure 2 As shown, the key point position determination method provided in this embodiment may include the following steps:

[0051] Step S201, obtaining the predicted position and confidence level of the target key point in the current video frame in the video to be processed.

[0052] Specifically, target key point detection can be performed on each video frame of the processed video in turn based on the position detection algorithm, so as to obtain the detection results of each video frame. When determining the target position of the target key point in the current video frame, it is possible to first determine whether the current video frame has the target key point based on the detection result corresponding to the current video frame, and determine whether the previous video frame has the target key point. Based on the judgment results, different strategies are adopted to determine the target position of the target key point in the previous video frame. This embodiment is aimed at the situation where the target key point exists in both the current video frame and the previous video frame.

[0053] Step S202, obtaining the target position of the target key point in the previous video frame.

[0054] Specifically, the target position of each target key point in each video frame may be determined in sequence according to the time sequence of the video frames, and the confidence level of the target position of each target key point in each video frame may also be determined.

[0055] In one embodiment, step S202 may specifically include: obtaining the target position and confidence level of the target key point in the previous video frame.

[0056] Step S202 and step S201 may be executed in parallel (eg Figure 2 As shown), step S202 may also be performed before or after step S201.

[0057] Optionally, if the target key point does not exist in the previous video frame, the predicted position of the target key point in the current video frame is directly determined as the target position of the target key point in the current video frame, and the confidence of the predicted position of the target key point in the current video frame is determined as the confidence of the target position of the target key point in the current video frame.

[0058] Step S203, calculating the position deviation between the predicted position of the target key point in the current video frame and the target position of the target key point in the previous video frame.

[0059] The position deviation is used to characterize the deviation between two positions and can be represented by the Euclidean distance corresponding to the two positions.

[0060] Specifically, the predicted position P′ of the target key point in the current video frame (the Nth video frame) N The target position P of the target key point in the previous video frame (the N-1th video frame) N-1 Position deviation ΔP N It can be expressed as: |P′ N P N-1 |.

[0061] Step S204, if the position deviation is greater than the first threshold, determining the target position of the target key point in the current video frame according to the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame and its confidence.

[0062] The first threshold may be a fixed threshold or a dynamically changing threshold.

[0063] Specifically, the first threshold may be determined according to the type of the target object in the video to be processed.

[0064] Specifically, the first threshold may be determined according to an image size of a target object corresponding to the target key point in the current video frame.

[0065] Specifically, when the position deviation ΔP NWhen it is greater than the first threshold a, it indicates that the deviation between the positions of the target key points in the two adjacent video frames, that is, the current video frame and the previous video frame, is large. Then, the target position of the target key point in the current video frame can be determined based on the weighted average of the positions of the target key points in the two adjacent video frames, thereby improving the accuracy of key position detection.

[0066] Specifically, when the position deviation ΔP N When it is greater than the first threshold a, the target position of the target key point in the current video frame can be determined based on the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame and its confidence.

[0067] Furthermore, when the position deviation is greater than a first threshold, a weighting coefficient can be determined based on the position deviation and / or the confidence of the predicted position of the target key point in the current video frame, and the target position of the target key point in the current video frame can be determined based on the weighting coefficient, the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame.

[0068] Specifically, when the position deviation is greater than the first threshold, the target position P of the target key point in the Nth video frame N The expression is:

[0069] P N =α×P′ N +(1-α)×P N-1

[0070] Wherein, α is the above-mentioned weighting coefficient, which is between 0 and 1.

[0071] Specifically, the weighting coefficient α may be determined according to the confidence of the predicted position of the target key point in the current video frame. Alternatively, the weighting coefficient α may be determined according to the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame. Alternatively, the weighting coefficient α may be determined according to the position deviation.

[0072] Exemplarily, the confidence level of the predicted position of the target key point in the current video frame may be directly determined as the weighting coefficient α.

[0073] Exemplarily, the weighting coefficient α can be determined based on the ratio of the confidence of the predicted position of the target key point in the current video frame to the confidence of the target position of the target key point in the previous video frame, that is, the first ratio, so that the ratio of α to 1-α is equal to the first ratio.

[0074] In one embodiment, the weighting coefficient α may be positively correlated with the position deviation, such as being proportional to the position deviation, that is, the larger the position deviation, the larger the weighting coefficient α.

[0075] Optionally, determining the target position of the target key point in the current video frame according to the target position of the target key point in the previous video frame and the predicted position and confidence level of the target key point in the current video frame includes:

[0076] According to the confidence C of the target position of the target key point in the previous video frame N-1 , the confidence C′ of the predicted position of the target key point in the current video frame N And the position deviation ΔP N , determine the weighting coefficient α; determine the target position of the target key point in the current video frame according to the weighting coefficient α, the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame.

[0077] Specifically, the confidence C of the target position of the target key point in the previous video frame can be based on the preset relationship N-1 , the confidence C′ of the predicted position of the target key point in the current video frame N And the position deviation ΔP N , determine the weighting coefficient α. In this preset relationship, the position deviation ΔP N The larger the value, the confidence C′ of the predicted position of the target key point in the current video frame N The higher the confidence C of the target position of the target key point in the previous video frame N-1 The lower it is, the larger the weighting coefficient α is.

[0078] Exemplarily, the preset relationship may be:

[0079]

[0080] Among them, P max To set the coefficient, it can be a fixed value, such as 100 pixels, 500 pixels, 1000 pixels, etc., or it can be determined based on the target object corresponding to the target key point; w1 and w2 are both coefficient constants so that α is between 0 and 1.

[0081] When the position deviations of adjacent frames are large, the weighting coefficients used in calculating the target positions of target key points in each video frame are dynamically adjusted based on confidence and position deviation, thereby achieving adaptive determination of the weighting coefficients and improving the accuracy of key point position determination.

[0082] Optionally, the method further includes:

[0083] The image size of the target object to which the target key point belongs in the current video frame is obtained; and the first threshold is determined according to the image size.

[0084] The image size of the target object may be represented by parameters such as the number of pixels, the number of columns, the number of rows, etc. included in the target object in the image (each video frame may be regarded as an image).

[0085] Specifically, if the image size of the target object in the current video frame is larger, the first threshold is larger. If the first threshold is determined to be the product of the image size and a preset coefficient, the preset coefficient may be 0.1, 0.2, 0.5 or other values.

[0086] Exemplarily, if the image size of the target object is 80 pixels, the first threshold may be 8 pixels.

[0087] Specifically, the image size of the target object can be determined by using a target object recognition algorithm to determine the outline of the target object, and then based on the outline of the target object, the image size of the target object can be determined.

[0088] Specifically, the image size of the target object may be determined based on the predicted positions of each preset key point of the target object in the current video frame.

[0089] The preset key points may be key points at both ends of the trunk or main body of the target object.

[0090] For example, taking a human body as an example, the preset key points may be key points on the neck and hips. Taking a face as an example, the preset key points may be key points on the forehead and chin. Taking a car as an example, the preset key points may be key points on the headlights and taillights of the car.

[0091] By dynamically setting the first threshold based on the image size of the target object, the accuracy of setting the first threshold is improved, and the low accuracy of determining the position of the key point due to the difference in the size of the target object is avoided, thereby avoiding obvious delay or jitter of the key point in the video.

[0092] Optionally, obtaining the image size of the target object to which the target key point belongs in the current video frame includes:

[0093] Acquire each first key point of a first part and each second key point of a second part of a target object to which the target key point belongs in a current video frame; determine the image size of the target object according to the position coordinates of each first key point and the position coordinates of each second key point.

[0094] The first part and the second part are both parts of the target object, and the first part and the second part can be located at two ends of the main body or trunk of the target object respectively.

[0095] For example, taking the human body as an example, the first part may be the neck, head, etc., and the second part may be the buttocks, legs, etc. Taking the face as an example, the first part may be the forehead, and the second part may be the chin. Taking a car as an example, the first part may be the headlight of the car, and the second part may be the taillight.

[0096] In one embodiment, the first key point and the second key point may both be the target key point mentioned above.

[0097] Specifically, based on the above position detection algorithm, the predicted positions, ie, position coordinates, of each first key point and each second key point in the current video frame may be determined.

[0098] Specifically, the image size of the target object may be determined as the distance between the average value of the position coordinates of each first key point in the current video frame and the average value of the position coordinates of each second key point in the current video frame.

[0099] Specifically, the position coordinates of the center of the first part can be determined according to the position coordinates of each first key point in the current video frame and the preset outer contour of the first part; the position coordinates of the center of the second part can be determined according to the position coordinates of each second key point in the current video frame and the preset outer contour of the second part; the image size of the target object can be determined according to the distance between the position coordinates of the center of the first part and the second part.

[0100] The image size of the target object is dynamically estimated by using the position coordinates of key points at two ends of the torso of the target object in the current video frame, thereby improving the accuracy of determining the image size and thus improving the accuracy of determining the first threshold.

[0101] Optionally, determining the first threshold according to the image size includes:

[0102] The first threshold is determined according to the image size and the part to which the target key point belongs.

[0103] In order to further improve the accuracy of determining the first threshold, when determining the first threshold, in addition to considering the image size of the target object in the current video frame, the first threshold may also be determined based on the location to which the target key point belongs or is located.

[0104] Specifically, the setting coefficient may be determined according to the part to which the target key point belongs, and the product of the image size and the setting coefficient may be determined as the first threshold.

[0105] Specifically, the setting coefficient can be determined according to the ratio of the size of the part to which the target key point belongs to and the size of the target object. The larger the ratio, the larger the corresponding setting coefficient. Different parts correspond to different setting coefficients.

[0106] For example, the setting coefficient corresponding to the eyes may be 0.01, the setting coefficient corresponding to the head may be 0.1, and the setting coefficient corresponding to the legs may be 0.5.

[0107] Exemplarily, the expression of the first threshold may be:

[0108] a=S N ×c0×c1

[0109] Among them, c0 is the setting constant; c1 is the setting coefficient; S N is the image size of the target object in the Nth video frame.

[0110] Step S205: If the position deviation is less than or equal to the first threshold, determining the target position of the target key point in the previous video frame as the target position of the target key point in the current video frame.

[0111] When the position deviation between two adjacent video frames is small, that is, the position deviation ΔP N Less than or equal to the first threshold value is less than or equal to the first threshold value a. Since the time difference between adjacent video frames is small, the target position of the target key point in the previous video frame can be directly used as the target position of the target key point in the current video frame, thereby avoiding jitter of the target key point.

[0112] After determining the target position of the target key point in the current video frame, in order to determine the target position of the target key point in the subsequent video frame, it is also necessary to determine the confidence of the target position of the target key point in the current video frame, that is, it is necessary to execute step S206.

[0113] Step S206, determining the confidence of the target position of the target key point in the current video frame as the product of the weighted average of the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame and the first coefficient.

[0114] The first coefficient β is greater than 1, and may be 1.1, 1.2 or other values, and the maximum value of the confidence level is 1 or 100%.

[0115] In one embodiment, step S206 may be performed before or after step S204 or step S205. Alternatively, step S204 or step S205 may be performed in parallel with step S206. Figure 2 shown.

[0116] Specifically, if the target key point is detected in two adjacent video frames or in two adjacent frames of the video to be processed, a higher confidence level can be determined as the confidence level of the target position of the target key point in the current video frame. That is, the confidence level C of the target position of the target key point in the current video frame is determined. N The first coefficient β and the confidence C of the target position of the target key point in the previous video frame are: N-1 and the confidence C′ of the predicted position of the target key point in the current video frame N The product of the weighted averages of N =β×(w3×C′ N +w4×C N-1 ), where w3 and w4 are the set weight coefficients, and their sum is 1.

[0117] In one embodiment, the weight coefficient of the confidence of the predicted position of the target key point in the current video frame, ie, w3, is greater than the weight coefficient of the confidence of the target position of the target key point in the previous video frame, ie, w4.

[0118] In one embodiment, the weight coefficient w3 may be the above-mentioned weight coefficient α, and correspondingly, the weight coefficient w4 is 1-α.

[0119] When there are target key points in two adjacent video frames, the confidence of the target position of the target key point in the latter video frame is calculated to achieve dynamic update of the confidence. The confidence is improved by the first coefficient, which increases the probability that the subsequent video frame uses the target position of the key point of the previous video frame, thereby reducing the situation of key point position jitter due to accidental errors in the detection algorithm itself and improving the accuracy of key point position detection.

[0120] After determining the target positions of each target key point in one of the current video frames of the video to be processed, the current video frame can be processed based on the target positions of each target key point in the current video frame, such as recognition processing, body beautification processing, and face beautification processing.

[0121] After determining the target position of each target key point in each video frame of the video to be processed, at least part of the video frames of the video to be processed may be processed based on the target position of each target key point in at least part of the video frames.

[0122] In this embodiment, for the case where the target key point is detected in two adjacent frames in the video to be processed, different position determination strategies are adopted based on the comparison result of the position deviation between the predicted position of the target key point in the current video frame and the target position of the target key point in the previous video frame and the first threshold value, that is, when the position deviation is greater than the first threshold value, the target position of the target key point in the current video frame is determined based on the weighted average of the predicted position of the target key point in the current video frame and the target position of the target key point in the previous video frame to improve the accuracy of key point position detection; when the position deviation is less than or equal to the first threshold value, the target position of the target key point in the current video frame is directly determined. The target position of the target key point in the previous video frame is used as the target position of the target key point in the current video frame to avoid key point jitter due to errors in the position detection algorithm; at the same time, the confidence of the predicted position of the target key point in the current video frame after increase is determined as the confidence of the target position of the target key point in the current video frame, thereby increasing the probability that subsequent video frames inherit or use the target position of the target key point in the current video frame, and through the dynamic update of the confidence, the accuracy of key point position detection is further improved, thereby effectively suppressing the jitter of the target key point in the processed video and improving the video processing quality.

[0123] Optionally, if the target key point does not exist in the current video frame, the method further includes:

[0124] Determine whether the confidence of the target position of the target key point in the previous video frame is greater than a second threshold; if it is greater than the second threshold, determine that the target position of the target key point in the previous video frame is the target position of the target key point in the current video frame; determine that the product of the confidence of the target position of the target key point in the previous video frame and the second coefficient is the confidence of the target position of the target key point in the current video frame, and the second coefficient is a positive number less than 1.

[0125] The second threshold b may be any positive number less than the maximum confidence value, such as 0.5, 0.6, 50%, 60%, etc. The second threshold may be a fixed value, and may also be determined according to the part or target object to which the target key point belongs.

[0126] Specifically, if the detection result indicates that there is a target key point in the previous video frame but not in the current video frame, it can be determined whether the confidence of the target position of the target key point in the previous video frame is high enough, that is, greater than the second threshold value b. If so, the target position of the target key point in the previous video frame is used as the target position of the target key point in the current video frame, and the reduced confidence of the target position of the target key point in the previous video frame is determined as the confidence of the target position of the target key point in the current video frame, that is, the confidence C of the target position of the target key point in the current video frame is determined.N is the confidence C of the target position of the second coefficient γ and the target key point in the previous video frame N-1 The product of N =γ×C N-1 .

[0127] When a target key point does not exist in the current video but exists in the previous video frame, the target position of the target key point in the previous video frame with high confidence is used to improve the continuity of the target key point in the video to be processed and avoid the target key point from appearing and disappearing at times. At the same time, by dynamically lowering the confidence, the possibility of subsequent video frames inheriting or using the target position of the target key point in the current video frame is weakened, thereby improving the accuracy of determining the position of the target key point, thereby improving the continuity of the movement of each key point (or target key point) in the video to be processed and improving the quality of video processing.

[0128] Optionally, if the detection result output by the position detection algorithm is that the target key point does not exist in the current video frame, and the confidence of the target position of the target key point in the previous video frame is less than or equal to the second threshold b, then it is determined that the target key point does not exist in the current video frame, and the target position and confidence of the target key point of the current video frame are cleared, such as setting the confidence to 0.

[0129] Optionally, if the detection result output by the position detection algorithm is that there is no target key point in the current video frame and there is no target key point in the previous video frame, it is determined that there is no target key point in the current video frame, and the target position and confidence of the target key point of the current video frame are cleared.

[0130] In one embodiment, after determining each target key point or the target position of each key point in each video frame in the video to be processed, a detection result can be generated based on the target position of each key point in each video frame, and the detection result can be output to a video processing module, such as a video processing application, a video processing applet, etc.

[0131] In one embodiment, various parts of the target object can also be processed according to the target position of each key point in each video frame, such as enlarging the eyes, lengthening the legs, slimming the waist, adding special effects to a certain part, etc., and the processed video to be processed can be saved and / or displayed, or the processed video to be processed can be sent to a user terminal or a terminal that uploads the video to be processed.

[0132] For example, Figure 3 Schematic diagram of the result of determining the position of key points of a human body provided in one embodiment of the present application, such as Figure 3As shown, the user can upload the video to be processed stored or shot by the user terminal to the preset application, and select the processing options to be performed on the video to be processed, such as body beautification, face beautification, and makeup, etc. The processing intensity can also be set by the slider above the "body beautification" option. The preset application or its server determines the key point target position for each frame of the video to be processed uploaded by the user based on the key point position determination method provided in the above embodiment. Figure 3 Take the user's selection of body beauty as an example. Figure 3 The solid dots represent the key points on the human body in the current video frame. Then, the preset application processes the human body based on the processing options and processing strength selected by the user for the target position of each key point on the human body. The body beautification processing may include lengthening the legs, slimming the waist, adjusting the head and neck ratio, etc.

[0133] Figure 4 A flowchart of a method for determining a key point position provided by another embodiment of the present application is shown in FIG. Figure 4 As shown, the key point position determination method mainly performs three steps in a traversal manner, namely: obtaining the current frame image, detecting the current frame image based on the position detection algorithm, so as to obtain the predicted position and confidence of each key point in the current frame image; smoothing the position of the key point, and outputting the final result, that is, determining the target position of the key point in the current frame image, and can also include determining the confidence of the target position of the key point in the current frame image. The specific determination process can refer to the relevant steps in the above embodiment, such as steps S102, S204, S205 and S206. For example, for each key point, based on the predicted position and confidence of the key point in the current frame image, and the target position and confidence of the key point in the previous frame image, the target position and confidence of the key point in the current frame image are determined. Then traverse to the next frame image, and execute the above three steps in a loop until the target position of the key point in all frames of the image to be processed is determined.

[0134] Figure 5 A flowchart of a method for determining a key point position provided by another embodiment of the present application is shown in FIG. Figure 5 As shown, the key point position determination method includes the following steps:

[0135] Step S501 : detecting the current video frame of the video to be processed based on a position detection algorithm.

[0136] The current video frame is not the first video frame of the video to be processed.

[0137] Step S502, determine whether the target key point is detected in the current video frame; if so, execute step S503; if not, jump to step S510.

[0138] Step S503, determine whether the target key point is detected in the previous video frame; if so, execute step S504; if not, jump to step S509.

[0139] Step S504, determining whether the position deviation between the two frames is greater than a first threshold; if so, executing step S505; if not, jumping to step S508.

[0140] The position deviation between the two frames is the position deviation between the prediction of the target key point in the current video frame and the target position of the target key point in the previous video frame.

[0141] Step S505 , based on the weighted position coordinates of the target key point between the two frames and the confidence level thereof, determine the target position and the confidence level of the target key point in the current video frame.

[0142] Specifically, the target position of the target key point in the current video frame may be determined according to a weighted average of the position coordinates of the target key point between two frames (such as the predicted position or the target position described above).

[0143] Specifically, the confidence of the target position of the target key point in the current video frame may be determined according to a weighted average of the confidences of the position coordinates of the target key point between two frames.

[0144] Furthermore, the weighted average of the confidences of the position coordinates of the target key point between the two frames can be determined, that is, the weighted average of the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame, multiplied by a coefficient greater than 1, that is, the first coefficient, is the confidence of the target position of the target key point in the current video frame.

[0145] Step S506, determining whether the current video frame is the last frame; if not, returning to step S501 to determine the target position of the target key point in the next video frame; if so, executing step S507.

[0146] Step S507: output the target position of the target key point in each video frame.

[0147] Step S508, continue to use the target position of the target key point in the previous video frame and increase the confidence level; continue to execute step S506.

[0148] Specifically, increasing the confidence level may be: determining the confidence level of the predicted position of the target key point in the current video frame after the increase or the confidence level of the target position of the target key point in the previous video frame after the increase as the confidence level of the target position of the target key point in the current video frame.

[0149] Furthermore, the weighted average of the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame can be determined, and the product of the confidence of the target position of the target key point in the current video frame and a coefficient greater than 1, i.e., the first coefficient or the third coefficient, is determined.

[0150] Step S509, based on the detection result of the current video frame, determine the target position and the confidence level of the target key point of the current video frame; return to step S506.

[0151] Specifically, the predicted position of the target key point in the current video frame and its confidence level can be directly determined as the target position of the target key point in the current video frame and its confidence level.

[0152] Step S510, determining whether the confidence of the target position of the target key point in the previous video frame is greater than a second threshold; if so, executing step S511, if not, executing step S512.

[0153] If there is no target key point in the previous video frame, the confidence of the target position of the target key point in the previous video frame can be determined to be 0 or a minimum value, that is, the confidence of the target position of the target key point in the previous video frame is determined to be less than the second threshold.

[0154] Step S511, continue to use the target position of the target key point in the previous video frame and reduce the confidence level; return to step S506.

[0155] The lowering of the confidence level is specifically as follows: determining the lowered confidence level of the target position of the target key point in the previous video frame as the confidence level of the target position of the target key point in the current video frame.

[0156] Specifically, the product of the confidence of the target position of the target key point in the previous video frame and a coefficient less than 1, ie, the second coefficient, may be determined as the confidence of the target position of the target key point in the current video frame.

[0157] Step S512, determining that there is no target key point in the current video frame, setting the target position and confidence level of the target key point in the current video frame to null, and returning to step S506.

[0158] Figure 6 A schematic diagram of a key point position determination device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the device includes: a predicted position acquisition module 610 and a target position determination module 620.

[0159] Among them, the predicted position acquisition module 610 is used to obtain the predicted position and confidence of the target key point in the current video frame in the video to be processed; the target position determination module 620 is used to determine the target position of the target key point in the current video frame according to the target position of the target key point in the previous video frame, the predicted position of the target key point in the current video frame and its confidence, so as to process the video to be processed based on the target position of the target key point.

[0160] Optionally, the target location determination module 620 includes:

[0161] A position deviation calculation unit is used to calculate the position deviation between the predicted position of the target key point in the current video frame and the target position of the target key point in the previous video frame; a first position determination unit is used to determine the target position of the target key point in the current video frame based on the target position of the target key point in the previous video frame, the predicted position of the target key point in the current video frame and its confidence level if the position deviation is greater than a first threshold.

[0162] Optionally, the first position determining unit is specifically configured to:

[0163] If the position deviation is greater than a first threshold, a weighting coefficient is determined based on the confidence of the target position of the target key point in the previous video frame, the confidence of the predicted position of the target key point in the current video frame, and the position deviation; the target position of the target key point in the current video frame is determined based on the weighting coefficient, the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame.

[0164] Optionally, the target location determination module 620 further includes:

[0165] The second position determination unit is used to determine the target position of the target key point in the previous video frame as the target position of the target key point in the current video frame if the position deviation is less than or equal to the first threshold.

[0166] Optionally, the device further comprises:

[0167] A confidence determination module is used to determine the confidence of the target position of the target key point in the current video frame after determining the target position of the target key point in the current video frame, which is the product of the weighted average of the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame and a first coefficient, wherein the first coefficient is greater than 1.

[0168] Optionally, the device further comprises:

[0169] The image size determination module is used to obtain the image size of the target object to which the target key point belongs in the current video frame; the first threshold determination module is used to determine the first threshold according to the image size.

[0170] Optionally, an image size determination module is specifically used to:

[0171] Acquire each first key point of a first part and each second key point of a second part of a target object to which the target key point belongs in a current video frame; determine the image size of the target object according to the position coordinates of each first key point and the position coordinates of each second key point.

[0172] Optionally, the first threshold determination module is specifically configured to:

[0173] The first threshold is determined according to the image size and the part to which the target key point belongs.

[0174] Optionally, the target location determination module 620 is further configured to:

[0175] If the target key point does not exist in the current video frame, determine whether the confidence of the target position of the target key point in the previous video frame is greater than a second threshold; if it is greater than the second threshold, determine that the target position of the target key point in the previous video frame is the target position of the target key point in the current video frame; determine that the product of the confidence of the target position of the target key point in the previous video frame and the second coefficient is the confidence of the target position of the target key point in the current video frame, and the second coefficient is a positive number less than 1.

[0176] The key point position determination device provided in the embodiment of the present application can be used to perform the above Figure 1 , Figure 2 , Figure 4 and Figure 5 The technical solutions provided in any corresponding embodiment have similar implementation principles and technical effects, which will not be described in detail in this embodiment.

[0177] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the electronic device provided in this embodiment includes:

[0178] At least one processor 710; and a memory 720 communicatively connected to the at least one processor; wherein the memory 720 stores computer-executable instructions; and the at least one processor 710 executes the computer-executable instructions stored in the memory so that the electronic device performs a method as provided in any of the foregoing embodiments.

[0179] Optionally, the memory 720 may be independent or integrated with the processor 710 .

[0180] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0181] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the method provided in any of the aforementioned embodiments can be implemented.

[0182] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any of the above embodiments when executed by a processor.

[0183] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0184] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0185] It should be understood that the above-mentioned processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and can also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0186] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0187] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0188] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0189] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods provided in each embodiment of the present application.

[0191] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0192] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining a key point position, characterized in that: include: Obtain the predicted position and confidence of the target key point in the current video frame in the video to be processed; Calculating a position deviation between a predicted position of the target key point in the current video frame and a target position of the target key point in the previous video frame; If the position deviation is greater than a first threshold, determining a weighting coefficient according to the confidence of the target position of the target key point in the previous video frame, the confidence of the predicted position of the target key point in the current video frame, and the position deviation; According to the weighting coefficient, the target position of the target key point in the previous video frame and the predicted position of the target key point in the current video frame, the target position of the target key point in the current video frame is calculated, so as to process the video to be processed based on the target position of the target key point.

2. The method according to claim 1, characterized in that: The method further comprises: If the position deviation is less than or equal to the first threshold, the target position of the target key point in the previous video frame is determined to be the target position of the target key point in the current video frame.

3. The method according to claim 1 or 2, characterized in that: After determining the target position of the target key point in the current video frame, the method further includes: The confidence of the target position of the target key point in the current video frame is determined as the product of a weighted average of the confidence of the predicted position of the target key point in the current video frame and the confidence of the target position of the target key point in the previous video frame and a first coefficient, where the first coefficient is greater than 1.

4. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtaining the image size of the target object to which the target key point belongs in the current video frame; The first threshold is determined according to the image size.

5. The method according to claim 4, characterized in that Get the image size of the target object to which the target key point belongs in the current video frame, including: Acquire each first key point of a first part and each second key point of a second part of the target object to which the target key point belongs in the current video frame; The image size of the target object is determined according to the position coordinates of each first key point and the position coordinates of each second key point.

6. The method according to claim 4, characterized in that Determining the first threshold according to the image size includes: The first threshold is determined according to the image size and the part to which the target key point belongs.

7. The method according to claim 1 or 2, characterized in that: If the target key point does not exist in the current video frame, the method further includes: Determine whether the confidence level of the target position of the target key point in the previous video frame is greater than a second threshold; If it is greater than the second threshold, determining the target position of the target key point in the previous video frame as the target position of the target key point in the current video frame; Determine the product of the confidence of the target position of the target key point in the previous video frame and the second coefficient as the confidence of the target position of the target key point in the current video frame, where the second coefficient is a positive number less than 1.

8. A key point position determination device, characterized in that: include: A predicted position acquisition module is used to obtain the predicted position and confidence level of the target key point in the current video frame in the video to be processed; The target position determination module includes a position deviation calculation unit for calculating the position deviation between the predicted position of the target key point in the current video frame and the target position of the target key point in the previous video frame; The target position determination module includes a first position determination unit, which is used to determine a weighting coefficient based on the confidence of the target position of the target key point in the previous video frame, the confidence of the predicted position of the target key point in the current video frame, and the position deviation if the position deviation is greater than a first threshold; calculate the target position of the target key point in the current video frame based on the weighting coefficient, the target position of the target key point in the previous video frame, and the predicted position of the target key point in the current video frame, so as to process the video to be processed based on the target position of the target key point.

Images