Attitude prediction method and device, computer equipment and storage medium

Abstract

The invention relates to a posture prediction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: predicting position information of a key point of a target object in a to-be-predicted image through a prediction model to obtain target position information of the key point; identifying attitude information of the target object based on the target position information; by adopting the method, the attitude of the target object can be accurately predicted.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a posture prediction method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] As an important research direction in the field of computer vision, posture prediction is widely used in the fields of activity analysis of various objects, intelligent video surveillance, etc., and has attracted the attention of many people.

[0003] At present, taking human posture prediction as an example, human posture prediction is mainly achieved through model training and application. Specifically, in the process of training the human posture prediction model, the mean square error is used to measure the loss between the predicted target point coordinates and the actual target point coordinates, so as to obtain the trained human posture prediction model; and based on the trained human posture prediction model, the human posture is predicted.

[0004] However, current posture prediction methods suffer from insufficient accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate posture prediction method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems.

[0006] In a first aspect, the present application provides a method for training a prediction model, the method comprising:

[0007] Identify a reference directed line segment formed by key points of the target object in the sample image, and determine the angle between the reference directed line segment and a preset direction line segment to obtain a reference angle;

[0008] Predicting the sample image to obtain the first predicted position of the key point of the target object;

[0009] Determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to the angle between the predicted directed line segment and the preset direction line segment;

[0010] Determining a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0011] Determining a second loss value according to the reference angle and the first predicted angle;

[0012] The prediction model is trained based on the first loss value and the second loss value.

[0013] In a second aspect, the present application also provides a prediction model training device, the device comprising:

[0014] An angle acquisition module, used to identify the reference directed line segment formed between the key points of the target object in the sample image, determine the angle between the reference directed line segment and the preset direction line segment, and obtain the reference angle;

[0015] A position prediction module, used to predict the sample image and obtain a first predicted position of a key point of the target object;

[0016] An angle prediction module, used to determine a predicted directed line segment according to a first predicted position of the key point, and determine a first predicted angle according to an angle between the predicted directed line segment and a preset direction line segment;

[0017] A first loss module, used to determine a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0018] A second loss module, used for determining a second loss value according to a reference angle and a first predicted angle;

[0019] The training module is used to train the prediction model according to the first loss value and the second loss value.

[0020] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Identify a reference directed line segment formed by key points of the target object in the sample image, and determine the angle between the reference directed line segment and a preset direction line segment to obtain a reference angle;

[0022] Predicting the sample image to obtain the first predicted position of the key point of the target object;

[0023] Determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to the angle between the predicted directed line segment and the preset direction line segment;

[0024] Determining a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0025] Determining a second loss value according to the reference angle and the first predicted angle;

[0026] The prediction model is trained based on the first loss value and the second loss value.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0028] Identify a reference directed line segment formed by key points of the target object in the sample image, and determine the angle between the reference directed line segment and a preset direction line segment to obtain a reference angle;

[0029] Predicting the sample image to obtain the first predicted position of the key point of the target object;

[0030] Determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to the angle between the predicted directed line segment and the preset direction line segment;

[0031] Determining a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0032] Determining a second loss value according to the reference angle and the first predicted angle;

[0033] The prediction model is trained based on the first loss value and the second loss value.

[0034] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0035] Identify a reference directed line segment formed by key points of the target object in the sample image, and determine the angle between the reference directed line segment and a preset direction line segment to obtain a reference angle;

[0036] Predicting the sample image to obtain the first predicted position of the key point of the target object;

[0037] Determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to the angle between the predicted directed line segment and the preset direction line segment;

[0038] Determining a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0039] Determining a second loss value according to the reference angle and the first predicted angle;

[0040] The prediction model is trained based on the first loss value and the second loss value.

[0041] The training method, apparatus, computer equipment, storage medium and computer program product of the above-mentioned prediction model can also predict the sample image after obtaining the reference angle between the reference directed line segment formed between the key points of the target object in the real sample image and the preset direction line segment, and obtain the first predicted position of the key point of the target object, and then determine the first predicted angle between the predicted directed line segment and the preset direction line segment based on the first predicted position. By obtaining four parameters, namely the first predicted position, the reference angle, the first predicted angle, and the annotated position of the key point of the target object, it is possible to pay attention not only to the first loss value determined by the real position and the predicted position, but also to the second loss value determined by the reference angle and the predicted angle. That is, starting from the two loss value dimensions of position and angle, instead of taking the single position loss as the loss of the model, the prediction model is comprehensively and accurately trained to obtain a prediction model that can be used for accurate prediction.

[0042] In a sixth aspect, the present application also provides a posture prediction method, the method comprising:

[0043] The prediction model is used to predict the position information of the key points of the target object in the image to be predicted, and the target position information of the key points is obtained;

[0044] Based on the target position information, identify the posture information of the target object;

[0045] The prediction model is trained according to the prediction model training method.

[0046] In a seventh aspect, the present application also provides a posture prediction device, the device comprising:

[0047] A posture prediction module is used to predict the position information of the key points of the target object in the image to be predicted through a prediction model to obtain the target position information of the key points;

[0048] The posture recognition module is used to recognize the posture information of the target object based on the target position information; wherein the prediction model is obtained according to the training method of the prediction model.

[0049] In an eighth aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0050] The prediction model is used to predict the position information of the key points of the target object in the image to be predicted, and the target position information of the key points is obtained;

[0051] Based on the target position information, identify the posture information of the target object;

[0052] The prediction model is trained according to the prediction model training method.

[0053] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0054] The prediction model is used to predict the position information of the key points of the target object in the image to be predicted, and the target position information of the key points is obtained;

[0055] Based on the target position information, identify the posture information of the target object;

[0056] The prediction model is trained according to the prediction model training method.

[0057] In a tenth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0058] The prediction model is used to predict the position information of the key points of the target object in the image to be predicted, and the target position information of the key points is obtained;

[0059] Based on the target position information, identify the posture information of the target object;

[0060] The prediction model is trained according to the prediction model training method.

[0061] The above-mentioned posture prediction method, device, computer equipment, storage medium and computer program product predict the position information of the key points of the target object in the image to be predicted through the prediction model to obtain the target position information of the key points; based on the target position information, the posture information of the target object is identified. In the whole process, the position of the key points of the target object in the image to be predicted is accurately predicted through the prediction model that can accurately predict, so as to accurately identify the posture of the target object in the image to be predicted based on the target position information of the key points. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0063] Figure 1 A diagram showing an application environment of a posture prediction method in an embodiment;

[0064] Figure 2 A flowchart of a method for training a prediction model in one embodiment;

[0065] Figure 3 is a schematic diagram of key points in a human body image in a specific application example;

[0066] Figure 4 is a schematic diagram of directed line segments of a right upper arm and a right forearm in a human body image in a specific application example;

[0067] Figure 5 A schematic diagram of a flow chart of a posture prediction method in one embodiment;

[0068] Figure 6 A flowchart of a prediction model training phase for human posture prediction in a specific application example;

[0069] Figure 7 is a structural block diagram of a training device for a prediction model in one embodiment;

[0070] Figure 8 is a structural block diagram of a posture prediction device in one embodiment;

[0071] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0073] As an important research direction in the field of computer vision and a key issue in the field of somatosensory technology, posture prediction is widely used in intelligent video surveillance, image detection and other fields. Taking human posture prediction as an example, specifically, human posture prediction is the task of locating human joints (such as shoulders, elbows, wrists, etc.) or human skeletons (such as lower limbs, upper arms, etc., skeletons are also joint lines) from monocular images or videos. Human posture prediction has become an important basis for applications such as human action recognition, human-computer interaction, and animation.

[0074] At present, posture prediction is generally achieved through residual networks and convolutional networks, that is, the image that needs to be predicted is processed through the backbone network of the residual network to obtain deep low-resolution features, and then the low-resolution features are processed through three cascaded convolutional networks to restore the high-resolution features of the image. High-resolution features are often represented in the form of heat maps, that is, the position of the estimated point representing the posture is displayed as 1, and the other points are displayed as 0.

[0075] However, some estimated points representing posture are not stable enough and are prone to jitter. For example, when the posture prediction image is a posture prediction image of a human body, the estimated points of the limbs are prone to jitter, resulting in obvious outliers in posture prediction. Therefore, in order to solve this problem, it is often necessary to use deep learning technology to calculate the loss during posture prediction.

[0076] In the process of deep learning, when training the prediction model based on the heat map, the loss between the mean square error prediction heat map and the target heat map is generally used. However, the loss calculated in this way only focuses on the error between the predicted point and the true value corresponding to the predicted point, and is insensitive to the direction of the key point deviation. Taking the prediction of human posture as an example, it cannot reflect the deviation of the limb angle. For example, the position error of the limb points is the same, but it may be affected by the angle error of the limbs, resulting in a larger gap. Therefore, the current posture prediction method lacks a certain degree of accuracy.

[0077] To this end, the present application provides a posture prediction method, including a prediction model training phase and a posture prediction phase.

[0078] In the training stage of the prediction model, the reference directed line segment formed by the key points of the target object in the sample image is identified, and the angle between the reference directed line segment and the preset direction line segment is determined to obtain the reference angle; the sample image is predicted to obtain the first predicted position of the key point of the target object; the predicted directed line segment is determined according to the first predicted position of the key point, and the first predicted angle is determined according to the angle between the predicted directed line segment and the preset direction line segment; the first loss value is determined according to the first predicted position and the marked position of the key point of the target object; the second loss value is determined according to the reference angle and the first predicted angle; the prediction model is trained according to the first loss value and the second loss value, that is, not only the first loss value determined by the real position and the predicted position is paid attention to, but also the second loss value determined by the reference angle and the predicted angle is paid attention to, that is, starting from the two loss value dimensions of position and angle, instead of taking the single position loss as the loss of the model, the prediction model is comprehensively and accurately trained to obtain a prediction model that can be used for accurate prediction.

[0079] In the posture prediction stage, the position information of the key points of the target object in the image to be predicted is predicted through the prediction model to obtain the target position information of the key points; based on the target position information, the posture information of the target object is identified; wherein the prediction model is trained according to the training method of the prediction model. In the whole process, the posture of the target object in the image to be predicted is accurately predicted through the prediction model that can support accurate prediction.

[0080] The posture prediction method provided in the embodiment of the present application includes a prediction model training phase and a posture prediction phase. The posture prediction method can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0081] In the training phase of the prediction model, the user interacts with the data acquisition interface of the terminal 102 and uploads a sample image to the terminal 102, wherein a number of key points are set in the sample image. The terminal 102 forwards the sample image to the server 104, and the server 104 identifies the reference directed line segment formed by the key points of the target object in the sample image; and determines the angle between the reference directed line segment and the preset direction line segment to obtain the reference angle; predicts the sample image to obtain the first predicted position of the key point of the target object; determines the predicted directed line segment according to the first predicted position of the key point, and determines the first predicted angle according to the angle between the predicted directed line segment and the preset direction line segment; determines the first loss value according to the first predicted position and the marked position of the key point of the target object; determines the second loss value according to the reference angle and the first predicted angle; and trains the prediction model according to the first loss value and the second loss value. The server 104 can also store the trained prediction model in a database.

[0082] When the user wants to predict the posture of the target object in a certain image, the posture prediction stage is entered. In the posture prediction stage, the user uploads the image to be predicted on the human posture estimation interface of the terminal 102. The terminal 102 generates a posture prediction request based on the image to be predicted, and sends the posture prediction request to the server 104. The server 104 obtains the posture prediction request and extracts the image to be predicted in the posture prediction request. The trained prediction model is obtained from the database. The position information of the key points of the target object in the image to be predicted is predicted through the prediction model to obtain the target position information of the key points; based on the target position information, the posture information of the target object is identified. Further, the server 104 can also push the posture information of the target object to the terminal 102, and the terminal 102 displays it to the user. In addition, the terminal 102 can also collect the image to be predicted through its own data acquisition component, and upload the collected image to be predicted to the server 104. The server 104 obtains the trained prediction model from the database or the local terminal to complete the posture prediction of the image to be predicted.

[0083] In other embodiments, the posture prediction method may also be applied to a terminal or a server alone to complete posture prediction.

[0084] For example, in the training stage of the prediction model, the user interacts with the data acquisition interface of the terminal and uploads a sample image to the terminal, wherein a number of key points are set in the sample image. The terminal trains the prediction model based on the sample image using the training method of the above prediction model. The trained prediction model is stored locally in the terminal. In the posture prediction stage, the user uploads the image to be predicted on the posture prediction interface of the terminal, the terminal obtains the image to be predicted, and obtains the prediction model from the local terminal. Through the prediction model, the posture prediction method is adopted to obtain the posture information of the target object in the image to be predicted. Furthermore, the terminal can also display the posture information of the target object in the image to be predicted to the user.

[0085] For another example, in the training phase of the prediction model, the server obtains sample images from the database, and based on the sample images, uses the above prediction model training method to train the prediction model. The trained prediction model is stored in the server database. In the posture prediction phase, the server obtains the image to be predicted and the prediction model from the server database, and uses the prediction model to obtain the posture information of the target object in the image to be predicted using the above posture prediction method.

[0086] In an exemplary embodiment, Figure 2 As shown, a prediction model training method is provided, which is applied to Figure 1 The server 104 in FIG. 1 is used as an example for explanation. Among them:

[0087] S100, identifying a reference directed line segment formed by key points of a target object in a sample image, and determining an angle between the reference directed line segment and a preset direction line segment to obtain a reference angle.

[0088] The sample images are images selected from images containing the target object for training the prediction model. The target object can be a human object or a non-human object, for example, a human or an animal.

[0089] There are several key points distributed on the target object. Key points refer to the joint points of the target object that have a greater impact on the posture prediction of the target object. In layman's terms, key points are more important joint points in the target object. For example, key points can be joint points such as the nose, left eye, right eye, left elbow, and right elbow. In addition, key points include key points of the trunk and key points of the limbs. A key point can be only a key point of the trunk, or only a key point of the limbs, or both a key point of the trunk and a key point of the limbs, that is, a key point at the connection between the trunk and the limbs.

[0090] The reference directed line segment is determined based on the key points. It is a connecting line segment between two specific key points. The reference directed line segment has directionality. The reference angle is the angle between the reference directed line segment and the preset direction line. The preset direction line segment is a line segment with a defined direction. The defined direction includes but is not limited to any direction such as vertical upward direction, horizontal right direction, northeast 45° direction, etc. The direction of the preset direction line segment can be defined by the user, but the definition needs to follow a unified rule, that is, the direction setting rules for each preset direction line segment need to be consistent.

[0091] Specifically, after acquiring the sample image, the server can first determine the target object in the sample image, and then identify the key points on the target object. Since the angle factor between the key points must be taken into account, the reference directed line segment formed by the combination of two key points can be identified based on the key points on the target object. Then, the angle between the reference directed line segment and the preset direction line segment is determined to obtain the reference angle. For example, the angle between the reference directed line segment and the vertical upward direction line segment is determined as the reference angle corresponding to the reference directed line segment; for another example, the angle between the reference directed line segment and the horizontal rightward direction line segment is determined as the reference angle corresponding to the reference directed line segment, etc.

[0092] Furthermore, the step of determining the angle between the reference directed line segment and the preset direction line segment includes: using trigonometric function theorem to determine the angle between the reference directed line segment and the preset direction line segment. More specifically, the starting position and the end position of the reference directed line segment are obtained, and based on the starting position and the end position of the reference directed line segment, the trigonometric function theorem is used to determine the angle between the reference directed line segment and the preset direction line segment.

[0093] Taking the preset direction line segment as the horizontal right direction line segment as an example, the starting position (1, 5) of the reference directed line segment, the end position (4, 2) of the reference directed line segment, the starting position (1, 5) of the horizontal right direction line segment, and the end position (4, 5) of the horizontal right direction line segment are obtained. Based on the starting position and the end position of the reference directed line segment, the length of the reference directed line segment is obtained; based on the position of the starting point of the horizontal right direction line segment and the position of the end point of the horizontal right direction line segment, the length of the horizontal right direction line segment is determined, and the angle between the reference directed line segment and the horizontal right direction line segment is obtained by using the trigonometric function relationship based on the length of the reference directed line segment and the length of the horizontal right direction line segment, and the obtained angle is determined as the reference angle.

[0094] In one embodiment, the sample image may include not only the target object but also other objects. Therefore, taking a person as an example, the sample image may be a single-person image or a multi-person image.

[0095] In the case where the sample image is a single-person image, the target object in the single-person image is directly determined, and then the reference directed line segments formed by the key points of the target object in the sample image are identified; in the case where the sample image is a multi-person image, that is, when there are multiple objects in one image, the target object in the multi-person image needs to be extracted to avoid the influence of other objects on the training process. For example, a target detector can be used to determine the target object from the multi-person image, and image extraction is performed on the determined target object, so that the extracted target object is used as an updated sample image, and the reference directed line segments formed by the key points of the target object are identified from the updated sample image.

[0096] In a specific application, there is more than one key point, including but not limited to: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0097] S200, predicting the sample image to obtain a first predicted position of a key point of the target object.

[0098] The first predicted position of the key point refers to the predicted position of the key point of the target object in the sample image after the sample image is predicted.

[0099] Specifically, the sample image is predicted to obtain the first predicted position of the key point of the target object. The method of predicting the sample image includes but is not limited to: predicting the sample image using historical experience or predicting the sample image using a deep learning method.

[0100] Taking the prediction of sample images based on historical experience as an example, we can obtain images of the same target object that include the target object itself within a preset historical time period, and based on the images that include the target object itself within the preset historical time period, predict the first predicted position of the key point of the target object in the sample image within the current time period.

[0101] Taking the prediction of sample images using deep learning methods as an example, the prediction model used in the deep learning process can be stored in a database in advance, and the prediction model is obtained from the database in response to the request carrying the sample image sent by the terminal. The prediction model is used to predict the sample image, and the sample image is processed after forward propagation of the prediction model to obtain the first predicted position of the key point of the target object in the sample image.

[0102] In one embodiment, the first predicted position of the key point includes but is not limited to being expressed in the form of coordinates. When the first predicted position of the key point is expressed in the form of coordinates, when obtaining the first predicted position of the key point, it can be regarded as taking the upper left corner of the sample image as the origin of the coordinate system for obtaining the position; in another embodiment, when obtaining the first predicted position of the key point, it can also be regarded as taking the lower left corner of the sample image as the origin of the coordinate system for obtaining the position. That is to say, when the first predicted position of the key point is expressed in coordinates, the origin of the coordinate system where the coordinates are located can be defined by the user, and then the first predicted position of the key point in the sample image is determined by a unified standard.

[0103] S300, determining a predicted directed line segment according to a first predicted position of the key point, and determining a first predicted angle according to an angle between the predicted directed line segment and a preset direction line segment.

[0104] Among them, each reference directed line segment corresponds to a predicted directed line segment one-to-one. Although both the reference directed line segment and the predicted directed line segment are directed line segments formed by connecting two key points in pairs, the reference directed line segment is a directed line segment formed by connecting two key points in pairs based on the real positions of the key points; while the predicted directed line segment is a directed line segment obtained by combining two key points in pairs based on the predicted positions of the key points. In order to improve the accuracy of training, the preset direction line segment needs to be consistent with the preset direction line segment when the reference angle is obtained. The first predicted angle is the angle between the predicted directed line segment and the preset direction line segment.

[0105] Specifically, after calculating the first predicted position of the key point of the target object, two specific key points are selected from a number of key points, and the selected key points are connected in a directed manner according to the first predicted position of the selected key points to obtain a predicted directed line segment. For example, the first predicted position of key point A is (1, 2), and the first predicted position of key point B is (3, 4). Using key point A as the starting point and key point B as the end point, (1, 2) and (3, 4) are connected to obtain a predicted directed line segment AB. Furthermore, in the process of determining the predicted directed line segment, the pairwise key point combination selected for determining the predicted directed line segment is consistent with the pairwise key point combination selected when forming the reference directed line segment.

[0106] Furthermore, the angle between the predicted directional line segment and the preset directional line segment is determined, and the first predicted angle is calculated according to the angle between the predicted directional line segment and the preset directional line segment.

[0107] The step of determining the angle between the predicted directed line segment and the preset direction line segment includes: using trigonometric function theorem to determine the angle between the predicted directed line segment and the preset direction line segment. More specifically, the starting position and the end position of the predicted directed line segment are obtained, and based on the starting position and the end position of the predicted directed line segment, the trigonometric function theorem is used to determine the angle between the predicted directed line segment and the preset direction line segment. The specific determination method is similar to the method of calculating the reference angle, which will not be repeated here.

[0108] It should be noted that the first predicted position and the first predicted angle at this time are not accurate, and the prediction model needs to be continuously updated based on the first predicted position and the first predicted angle so that the prediction model can output the accurate first predicted position of the key point of the target object in the sample image.

[0109] S400, determining a first loss value according to the first predicted position and the marked position of the key point of the target object.

[0110] Among them, the first predicted position is the predicted position of the key point of the target object, and the labeled position is the actual position of the key point of the target object. The loss is used to measure the degree of difference between the predicted value and the actual value of the model. For the model, the smaller the loss, the better the robustness of the model. The first loss value is the loss of the position of the key point of the target object, that is, the first loss value is obtained based on the difference between the predicted position and the actual position of the key point of the target object.

[0111] Specifically, since the first loss value is obtained based on the difference between the predicted position and the actual position of the key point of the target object, the difference between the predicted position and the actual position actually refers to the difference between the predicted position and the actual position. Therefore, by taking the difference between the first predicted position and the marked position of the key point of the target object, the difference between the first predicted position of the key point and the marked position of the key point can be quantified to obtain the first loss value.

[0112] S500, determining a second loss value according to a reference angle and a first predicted angle.

[0113] Among them, the second loss value is the loss of the angle between the directed line segment formed between the key points and the preset direction line segment, that is, the second loss value is obtained based on the difference between the reference angle and the first predicted angle.

[0114] Specifically, since the second loss value is obtained based on the difference between the reference angle and the first predicted angle, the difference between the reference angle and the first predicted angle actually refers to the difference between the reference angle and the first predicted angle. Therefore, by taking the difference between the reference angle and the first predicted angle, the difference between the reference angle and the first predicted angle can be quantified to obtain the second loss value.

[0115] S600: training a prediction model according to the first loss value and the second loss value.

[0116] Specifically, according to the first loss value and the second loss value, the prediction model is back-propagated and trained from the perspective of the losses in terms of position and angle. That is, according to the first loss value and the second loss value, the final loss is obtained, and based on the final loss, the prediction model is back-propagated and trained. The purpose of the back-propagation process can be to update the weight coefficient and bias term of the prediction model according to the final loss to minimize the final loss, thereby obtaining the prediction model with the minimum final loss; or it can be to continuously update the weight coefficient and bias term of the prediction model according to the final loss until the preset total number of training times is reached, thereby obtaining the prediction model trained in the training process corresponding to the preset total number of training times.

[0117] It should be explained that the final loss is essentially the updated first loss value, that is, after determining the first loss value based on the first predicted position and the labeled position of the key point of the target object, it is necessary to optimize the determined first loss value based on the second loss value, that is, with the assistance of the angle loss, to obtain the optimized first loss value, that is, the final loss. In other words, in the application process of the model, the model predicts more accurate key point position information after calibrating the angle loss.

[0118] In one embodiment, for any round of training process of the human body posture estimation model, the first loss value and the second loss value of each round of training process can be obtained by the above method, and then the prediction model is trained according to the first loss value and the second loss value, and then the prediction model trained in this round of training process is used to perform prediction processing for the next round of training process.

[0119] In one embodiment, the training end condition of the prediction model includes: the number of training times reaches a preset threshold, or the final loss obtained based on the first loss value and the second loss value tends to be stable in several consecutive training processes. The trained prediction model is used to accurately output the position of the key point of the target object in the image when the image is input.

[0120] In the training of the above-mentioned prediction model, after obtaining the reference angle between the reference directed line segment formed between the key points of the target object in the real sample image and the preset direction line segment, the sample image can also be predicted to obtain the first predicted position of the key point of the target object, and then based on the first predicted position, the first predicted angle between the predicted directed line segment and the preset direction line segment is determined. By obtaining four parameters, namely the first predicted position, the reference angle, the first predicted angle, and the labeled position of the key point of the target object, it is possible to pay attention not only to the first loss value determined by the real position and the predicted position, but also to the second loss value determined by the reference angle and the predicted angle. That is, starting from the two loss value dimensions of position and angle, instead of taking the single position loss as the loss of the model, the prediction model is comprehensively and accurately trained to obtain a prediction model that can be used for accurate prediction.

[0121] In one embodiment, identifying a reference directed line segment formed by key points of a target object in a sample image includes:

[0122] Obtain candidate directed line segments formed by key points; and screen reference directed line segments from the candidate directed line segments based on the connection relationship between the key points.

[0123] Among them, the connection relationship between key points refers to the joint connection relationship determined based on life experience. Taking the target object as the human body as an example, for example, after the key point of the left shoulder is connected with the key point of the left elbow, the right upper arm is formed, and the right upper arm satisfies the joint connection relationship of the human body. Therefore, the connection between the key point of the left shoulder and the key point of the left elbow satisfies the connection relationship between the key points; for another example, after the key point of the left ear is connected with the key point of the left hip, a joint cannot be formed, that is, the joint connection relationship is satisfied. Therefore, the connection between the key point of the left ear and the key point of the left hip does not satisfy the connection relationship between the key points.

[0124] Specifically, the key points are combined in pairs to obtain a number of candidate directed line segments. When determining the candidate directed line segments, the direction may be determined first, and then the candidate directed line segments are determined based on the direction, or the candidate line segments may be determined first, and then the direction of the candidate line segments is set.

[0125] For example, the process of first determining the direction and then determining the candidate directed line segments based on the direction includes: taking any key point as the starting key point, taking any key point other than the starting point as the ending key point, connecting the starting key point and the ending key point, and regarding the starting key point and the ending key point as a pair of key point combinations, then in the obtained candidate directed line segments, the key point combination of each candidate directed line segment cannot be the same as the key point combination of another candidate directed line segment, that is, in the case where the candidate directed line segment already has a starting key point of A and an ending key point of B, the candidate directed line segment cannot have a starting key point of B and an ending key point of A, nor can it have a starting key point of A and an ending key point of B. For another example, in the case where the key points include A, B, and C, the candidate directed line segments formed by the key points include: A→B, A→C, and B→C.

[0126] The process of first determining the candidate line segments and then setting the directions of the candidate line segments includes: performing non-repeating combinations between any two key points to obtain a number of candidate line segments, and setting the directions of each candidate line segment in turn to obtain candidate directed line segments.

[0127] Furthermore, since the obtained candidate directed line segments are not necessarily line segments that meet the connection relationship between the key points, it is necessary to screen the candidate directed line segments. The screening means include but are not limited to: screening the line segments that meet the connection relationship between the key points, that is, the target line segments that meet the real joint connection relationship, from the candidate directed line segments.

[0128] Furthermore, since the angle between the reference directed line segment and the preset direction line segment of the target object's torso is larger and more unstable than the angle between the reference directed line segment and the preset direction line segment of the target object's torso, the directed line segment that satisfies the connection relationship between the key points is selected from the candidate directed line segments, which may be the limb line segments that satisfy the connection relationship between the key points, for example, the limb line segments of the right upper arm are selected, the limb line segments of the left upper arm are selected, etc. The limb line segments that satisfy the connection relationship between the key points are selected and determined as reference directed line segments.

[0129] In an exemplary application, taking the target object as a human body, as Figure 3As shown, the human body includes the following 17 key points, specifically: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle. The process of identifying the reference directed line segments formed by the key points of the human body includes:

[0130] Obtain several candidate directed line segments formed between the above 17 key points. From the several candidate directed line segments, based on the connection relationship between the key points, screen out the limb line segments that satisfy the human joint connection relationship. For example, the limb line segments that satisfy the human joint connection relationship include: 5→7 right upper arm, 7→9 right forearm, 6→8 left upper arm, 8→10 left forearm, 11→13 right thigh, 13→15 right calf, 12→14 left thigh, 14→16 left calf. The screened limb line segments that satisfy the human joint connection relationship are determined as reference directed line segments.

[0131] In the above embodiment, by screening out limb line segments that satisfy the connection relationship between the key points from the candidate directed line segments formed from the key points, it is possible to pay attention to the influence of the angles corresponding to the limb line segments on the posture prediction, rather than just the influence of the positions of the key points on the posture prediction. Then, the posture prediction model can be trained comprehensively and accurately through the two dimensions of the angle between the limb directed line segments and the preset direction line segments and the positions of the key points.

[0132] In one embodiment, determining the angle between the reference directional line segment and the preset directional line segment to obtain the reference angle includes:

[0133] Identify the starting key point of the reference directed line segment; obtain the preset direction line segment starting from the starting key point; determine the angle between the reference directed line segment and the preset direction line segment to obtain the reference angle.

[0134] Each reference directed line segment is formed by connecting two key points. Therefore, each reference directed line segment includes two key points, namely, the start key point and the end key point. The start key point is the starting point of the reference directed line segment, and the end key point is the end point of the reference directed line segment. The direction of the preset direction segment is a custom direction, including but not limited to the horizontal right direction, the vertical upward direction, etc.

[0135] Specifically, in order to determine the reference angle more efficiently, the starting key point and the ending key point of the reference directed line segment can be identified, and a preset direction line segment can be set with the starting key point as the starting point. The reason for setting the directions of the reference directed line segment and the preset direction line segment is to make the angle between the reference directed line segment and the preset direction line segment unique.

[0136] Based on the angle between the reference directed line segment and the preset direction line segment, the reference angle can be determined. The reference angle is the angle between the reference directed line segment and the preset direction line segment.

[0137] like Figure 4 The diagram shows a separate schematic diagram of the right upper arm 5→7 and the right forearm 7→9. Taking the direction of the preset direction line segment as the horizontal right direction as an example, the starting point of the right upper arm is the key point 5, and the key point 5 is used as the starting point of the preset direction line segment, and a preset direction line segment of the horizontal right direction line segment is set. It can be seen that Figure 4 The preset direction line segment in is the horizontal line A, and the reference angle a1 between the horizontal line A and the right upper arm 5→7 is calculated clockwise. Similarly, the reference angle a2 between the horizontal line B and the right forearm 7→9 can also be calculated.

[0138] In this embodiment, by taking the starting key point of the reference directed line segment as the starting point and accurately setting the preset direction line segment, the reference angle can be efficiently obtained based on the angle between the reference directed line segment and the preset direction line segment. Furthermore, since the preset direction line segment and the reference directed line segment are both set with directions, the obtained reference angle is unique, and the reference angle can be accurately obtained.

[0139] In one embodiment, the key points include N; N is an integer greater than or equal to 2; each key point includes a first predicted position and a marked position; and determining a first loss value according to the first predicted position and the marked position of the key point of the target object includes:

[0140] Determine a first difference between a first predicted position and annotated position of each key point to obtain N first differences; and determine a first loss value based on a mean of the sum of squares of the N first differences.

[0141] Specifically, each key point includes a first predicted position and an annotated position, and a first loss value of the prediction model is calculated based on the first predicted positions and annotated positions of the N key points.

[0142] More specifically, the first predicted position and the marked position of each key point are obtained, and the difference calculation is performed on the first predicted position and the marked position of each key point to obtain the first difference between the first predicted position and the marked position of each key point. Based on the N first differences, the first loss value of the prediction model is calculated.

[0143] In the present application, the first loss value of the prediction model is calculated based on N first differences, and the MSE (Mean Square Error) calculation method is adopted, that is, the first loss value of the prediction model is determined based on the mean of the sum of the squares of the N first differences.

[0144] In one embodiment, determining the first loss value of the prediction model based on the mean of the sum of squares of the N first differences includes: determining the mean of the sum of squares of the N first differences as the first loss value of the prediction model.

[0145] For example, let the total number of key points be N, and for any key point i in the sample image, the first predicted position of key point i is f(x i ), marked as y i , the expression of the first loss value LOSS1 of the prediction model is:

[0146]

[0147] In another embodiment, the first loss value of the prediction model is determined based on the mean of the sum of squares of the N first differences, including: based on the mean of the sum of squares of the N first differences, scaling the calculated mean, that is, multiplying the mean by a scaling coefficient, wherein the scaling coefficient is a positive number, and based on the scaling coefficient, the mean is enlarged or reduced, and the scaled mean is determined as the first loss value of the prediction model.

[0148] In this embodiment, by adopting the mean square error, the first loss value of the sample image can be determined efficiently and accurately, and then the prediction model can be accurately trained by combining the first loss value and the second loss value.

[0149] In one embodiment, the reference angles include M; the first prediction angles include M; the first prediction angles correspond to the reference angles one by one; M is a positive integer; and determining the second loss value according to the reference angles and the first prediction angles includes:

[0150] For each of the M first prediction angles, determine the second difference between the first prediction angle and the corresponding reference angle to obtain M second differences; sum the M second differences to obtain a difference sum; when the absolute value of the difference sum is less than or equal to a preset difference, determine the second loss value based on the difference between the absolute value of the difference sum and the first preset value; when the absolute value of the difference sum is greater than the preset difference, multiply the square of the difference sum by the second preset value to determine the second loss value.

[0151] Among them, the preset difference, the first preset value, and the second preset value are all constant values ​​defined based on historical experience. The preset difference is used as a selection condition for the calculation formula under different circumstances of the second loss value. When the absolute value of the preset parameter is less than or equal to the preset difference, the first calculation formula is used to calculate the second loss value. When the absolute value of the preset parameter is greater than the preset difference, the second calculation formula is used to calculate the second loss value. The first preset value is a constant value in the first calculation formula, which is used to calculate the second loss value; the second preset value is a constant value in the second calculation formula, which is used to calculate the second loss value.

[0152] Specifically, the first prediction angles correspond to the reference angles one-to-one. When the number of reference angles is M, the number of first prediction angles is also M.

[0153] First, for each of the M first prediction angles, a second difference between the first prediction angle and the corresponding reference angle is determined to obtain M second differences, and further, the M second differences are summed to obtain a difference sum.

[0154] Secondly, the difference and is analyzed. When the difference and is less than or equal to the preset difference, the first calculation formula is used to calculate the second loss value. When the difference and is greater than the preset difference, the second calculation formula is used to calculate the second loss value. In one embodiment, the first calculation formula is a linear calculation formula to improve the problem of non-smooth zero point; the second calculation formula is a nonlinear calculation formula, so that when the difference and is large, it is not sensitive to abnormal values, so that the angle loss does not fluctuate too fast.

[0155] Specifically, when the absolute value of the sum of the differences is less than or equal to the preset difference, the second loss value is determined according to the difference between the absolute value of the sum of the differences and the first preset value. Determining the second loss value according to the difference between the absolute value of the sum of the differences and the first preset value includes: determining the difference between the absolute value of the sum of the differences and the first preset value as the second loss value. That is, the first calculation formula includes: LOSS2=|sum of differences|-first preset value, |sum of differences|>preset difference.

[0156] In addition, determining the second loss value according to the difference between the absolute value of the sum of the differences and the first preset value also includes: obtaining the difference between the absolute value of the sum of the differences and the first preset value, performing a scale transformation on the difference, and determining the scale-transformed value as the second loss value. The scale transformation includes reducing or enlarging the difference.

[0157] When the absolute value of the sum of the differences is greater than the preset difference, the second loss value is determined according to the product of the square of the sum of the differences and the second preset value. The second loss value is determined according to the product of the square of the sum of the differences and the second preset value, including: determining the product of the square of the sum of the differences and the second preset value as the second loss value, that is, the second calculation formula includes: .

[0158] In addition, determining the second loss value according to the product of the square of the sum of the differences and the second preset value also includes: obtaining the product of the square of the sum of the differences and the second preset value, scaling the product, and determining the scaled product as the second loss value. Scaling includes reducing or enlarging the product.

[0159] In a specific application, let the predicted directed line segment l j The first prediction angle is F(l j ), the reference angle corresponding to the first prediction angle is a j , then the second difference between the first predicted angle and the corresponding reference angle is F(l j )- a j . Further, the sum of the difference of the M second differences is calculated M (F(l j )- a j ). Let the preset difference be 1, the first preset value be 0.5, and the second preset value be 0.5. Based on the sum of the difference of the second difference, the formula for calculating the second loss value is as follows:

[0160] LOSS2=|Σ M (F(l j )- a j )|-0.5,|Σ(F(l j )- a j )|>1

[0161] LOSS2=0.5(Σ M (F(l j )- a j ) 2 ,|Σ(F(l j )- a j )|≤1

[0162] In this embodiment, by adopting two different calculation formulas, the angle loss is accurately calculated when the difference and the angle meet different conditions. In the above process of calculating the angle loss, by setting the first calculation formula as a linear calculation formula, the problem of zero point non-smoothness can be improved, and by setting the second calculation formula as a nonlinear calculation formula, when the difference and the angle are large, it is not sensitive to abnormal values, so that the angle loss will not fluctuate too fast.

[0163] In one embodiment, training a prediction model according to the first loss value and the second loss value includes:

[0164] When the current training round is L, obtain the angle loss weight coefficient of the second loss value in the Lth round of training; wherein L is an integer greater than 1 and less than or equal to C; C presets the total number of training times; based on the angle loss weight coefficient, perform weighted summation on the first loss value and the second loss value to obtain a third loss value; based on the third loss value, train the prediction model.

[0165] Among them, the angle loss weight coefficient is the proportion of the second loss value when training the prediction model based on the first loss value and the second loss value, and the angle loss weight coefficient is not fixed, and can change adaptively with the training process, that is, the angle loss weight coefficient is the angle loss function corresponding to the current training round, and the angle loss weight coefficient is different in different training rounds. The angle loss weight coefficient is a coefficient in the range of (0, 1). The third loss value is the first loss value after optimization based on the second loss value.

[0166] Specifically, considering that in the early stage of training, the prediction model is more sensitive to the deviation angle between the directed line segment formed by the key point and the preset direction line segment than the loss of the key point position. Therefore, the loss of the deviation angle will fluctuate greatly, resulting in a slow convergence speed in the early stage of model training. For this reason, it is proposed to use an adaptive loss weight model to solve this problem.

[0167] The adaptive loss weight model essentially sets an adaptive angle loss weight coefficient for the second loss value. During each round of training of the prediction model, the angle loss weight coefficient is not fixed, but will change adaptively with the change of training rounds.

[0168] According to the first loss value and the second loss value, the prediction model is trained, including: based on the angle loss weight coefficient, the first loss value, and the second loss value, the prediction model is trained. More specifically, based on the angle loss weight coefficient, the first loss value and the second loss value are weighted and summed to obtain a third loss value, and the prediction model is trained based on the third loss value. That is, the third loss value is calculated based on the first loss value and the product of the angle loss weight coefficient and the second loss value. For example, let the first loss value be LOSS1, the second loss value be LOSS2, and the angle loss weight coefficient be , then the third loss value .

[0169] In this embodiment, by obtaining the angle loss weight coefficient corresponding to the current training round, in the subsequent calculation of the third loss value used to train the model, the second loss value can be adaptively updated based on the increase in training rounds to obtain a more accurate third loss value, thereby accurately training the prediction model.

[0170] In one embodiment, obtaining the angle loss weight coefficient of the second loss value during the Lth round of training includes:

[0171] Obtain a first weight coefficient for the second loss value during the Lth round of training, and obtain a second weight coefficient for the second loss value during the L-1th round of training; obtain an angle loss weight coefficient for the second loss value during the Lth round of training based on the product of the first weight coefficient and the second weight coefficient.

[0172] The angle loss weight coefficient includes the product of the first weight coefficient and the second weight coefficient. The first weight coefficient is used to adjust the second loss value to solve the problem of large fluctuation of the second loss value in the early stage of model training; the second weight coefficient is used to adjust the fluctuation of the second loss value to solve the problem of instability of the calculated third loss value due to excessive fluctuation of the second loss value in model training.

[0173] Specifically, the angle loss weight coefficient of the second loss value in the Lth round of training is obtained, including the following three implementation methods:

[0174] The first method is to obtain a first weight coefficient of the second loss value during the L-th round of training; and determine the first weight coefficient of the second loss value during the L-th round of training as the angle loss weight coefficient of the second loss value during the L-th round of training.

[0175] In the first implementation, the first weight coefficient in the first round of training is set to the minimum weight coefficient, and as the number of training rounds increases, the first weight coefficient is gradually increased until it stabilizes. The purpose of this setting is to make the weight of the second loss value smaller in the early stage of model training, so that the model pays more attention to the first loss value of the key point to speed up the convergence of the model. After the first loss value of the key point converges to a certain degree, it gradually changes to the model training process dominated by the second loss value, and the convergence speed of the model in the later stage will also become faster.

[0176] The second method is to obtain the second weight coefficient of the second loss value in the L-1th round of training; and determine the second weight coefficient of the second loss value in the L-1th round of training as the angle loss weight coefficient of the second loss value in the Lth round of training.

[0177] In the second implementation method, in order to avoid excessive fluctuations in the second loss value and interfere with the model training process, the present application sets a second weight coefficient, and the second weight coefficient is related to the fluctuation of the second loss value. When the fluctuation of the second loss value in the current training process is too large, the second weight coefficient in the current training process will decrease accordingly. Conversely, when the fluctuation of the second loss value in the current training process is too small, the second weight coefficient in the current training process is a larger coefficient. Therefore, the second loss value can be maintained within a stable range to avoid interference with the model training process.

[0178] The third method is to obtain the first weight coefficient of the second loss value in the Lth round of training, and obtain the second weight coefficient of the second loss value in the L-1th round of training; based on the product of the first weight coefficient and the second weight coefficient, obtain the angle loss weight coefficient of the second loss value in the Lth round of training.

[0179] In the third implementation, in order to avoid the second loss value from fluctuating greatly in the early stage and affecting the convergence, two weight coefficients are set for the second loss value: the first weight coefficient , the second weight coefficient Let the first loss value be LOSS1, the second loss value be LOSS2, then the third loss value is .

[0180] Among them, the first weight coefficient gradually increases with the increase of model training rounds until it stabilizes. The first weight coefficient is used to reduce the weight of the second loss value, so that the model pays more attention to the first loss value of the key point. After the first loss value converges to a certain degree, it gradually changes to a model training process dominated by the second loss value to speed up the convergence of the model. In practical applications, there is a specific threshold for the first weight coefficient, and the first weight coefficient cannot exceed the threshold so that the second loss value can gradually stabilize.

[0181] The second weight coefficient can maintain the second loss value within a stable range to avoid interference with the model training process. When the fluctuation of the second loss value in the current training process is too large, the second weight coefficient in the current training process will decrease accordingly. Conversely, when the fluctuation of the second loss value in the current training process is small, the currently trained second weight coefficient is a larger coefficient.

[0182] The product of the first weight coefficient and the second weight coefficient is determined as the angle loss weight coefficient in the current round of training. By setting the first weight coefficient and the second weight coefficient for the second loss value, the model can converge normally, thereby improving the training effect of the model.

[0183] In this embodiment, by setting the first weight coefficient for the second loss value, the model training can focus more on the first loss value of the key point in the early stage, and focus more on the second loss value in the later stage, thereby speeding up the model training; by setting the second weight coefficient for the second loss value, the angle loss in the model training process can be kept within a stable range, and abnormalities are not likely to occur, so that the model can converge normally, thereby improving the model training effect.

[0184] In one embodiment, obtaining a first weight coefficient of a second loss value in the Lth round of training includes:

[0185] Determine the ratio between the number of training times in the Lth round of training and half of the total number of training times; when the ratio is less than a preset threshold, determine the ratio as a first weight coefficient for the second loss value in the Lth round of training; when the ratio is greater than or equal to the preset threshold, determine the preset threshold as the first weight coefficient for the second loss value in the Lth round of training.

[0186] The total number of training times is the total number of training times preset before the training process begins. When the total number of training times is reached, the training process of the model stops. The preset threshold is the maximum value of the first weight coefficient, and the first weight coefficient cannot be set to a value exceeding the preset threshold.

[0187] Specifically, determine the ratio of the number of training times in the Lth round of training to half of the total number of training times. For example, when the total number of training times is 100, when L is 30, the number of training times in the 30th round of training is 30, and the ratio of the number of training times in the 2nd round of training to half of the total number of training times is 3 / 5. For another example, when L is 60, the number of training times in the 60th round of training is 60, and the ratio of the number of training times in the 60th round of training to half of the total number of training times is 6 / 5.

[0188] When the ratio between the number of training times in the Lth round of training and half of the total number of training times is less than a preset threshold, the ratio is determined as a first weight coefficient of the second loss value in the Lth round of training. When the ratio between the number of training times in the Lth round of training and half of the total number of training times is greater than or equal to a preset threshold, the preset threshold is determined as a first weight coefficient corresponding to the current training round.

[0189] When the ratio between the number of training times in the Lth round of training and half of the total number of training times is determined as the first weight coefficient of the second loss value in the Lth round of training, since the ratio between the number of training times in the Lth round of training and half of the total number of training times increases with the increase of training rounds, the first weight coefficient also increases with the increase of training rounds, and since when the ratio is greater than or equal to the preset threshold, the preset threshold is determined as the first weight coefficient in the Lth round of training, that is, the preset threshold is the maximum value of the first weight coefficient, that is, the first weight coefficient increases with the increase of training rounds, and when the ratio reaches the preset threshold, the first weight coefficient tends to a stable state, that is, the first weight coefficient stabilizes at the preset threshold.

[0190] The ratio between the number of training times in the Lth round of training and half of the total number of training times may be greater than 1. Therefore, the preset threshold also needs to be greater than 1. In practical applications, the preset threshold is generally set to 1.2, and the preset threshold can be adjusted by the user according to the model training effect.

[0191] That is to say, let the number of training times in the Lth round of training be e, the total number of training times be epoch, and half of the total number of training times be , the ratio between the number of training times in the Lth round of training and half of the total number of training times is e / h, then the first weight coefficient =min(e / h, preset threshold). In a practical application, the first weight coefficient =min(e / h,1.2).

[0192] In this embodiment, by determining the smaller value of the ratio between the number of training times in the Lth round of training and half of the total number of training times and the preset threshold as the first weight coefficient, the first weight coefficient can be a value that increases with the increase in the number of training rounds and gradually tends to be stable. By setting the first weight coefficient, the impact of the second loss value on the model training in the early stage of model training can be reduced, thereby speeding up the efficiency of model training. And as the number of training times increases, the first weight coefficient is increased accordingly to increase the second loss value, so that the model training process gradually changes to a model training process dominated by the second loss value in the later stage of model training, thereby improving the accuracy and efficiency of model training.

[0193] In one embodiment, obtaining a second weight coefficient of a second loss value in the L-1th round of training includes:

[0194] Obtain the second predicted position of each key point in the L-1th round of training, and determine the position loss value of the key point in the L-1th round of training based on the second predicted position and the marked position; obtain the second predicted angle between each predicted directional line segment and the preset direction line segment in the L-1th round of training; determine the angle loss value in the L-1th round of training based on M second predicted angles and corresponding reference angles; obtain the second weight coefficient of the second loss value in the L-1th round of training based on the position loss value and the angle loss value.

[0195] Among them, the position loss value is the first loss value of the key point in the L-1 round of training. This is to distinguish it from the first loss value of the key point in the L round of training. Therefore, the position loss value is used to define it, and the angle loss value is the same. The second predicted position is the predicted position of the key point in the L-1 round of training; the marked position is the actual position of the key point; the second predicted angle is the angle between each predicted directed line segment and the preset direction line segment; the reference angle is the actual angle between the reference directed line segment and the preset direction line segment; the reference angle includes M; the predicted directed line segment corresponds to the reference directed line segment one-to-one; the second predicted angle corresponds to the reference angle one-to-one.

[0196] Specifically, the second weight coefficient of the second loss value used for training in the L-th round of training is obtained in the L-1th round of training. That is, in the L-1th round of training, the position loss value and the angle loss value in the L-1th round of training are obtained, and based on the position loss value and the angle loss value in the L-1th round of training, the second weight coefficient of the second loss value in the L-1th round of training is obtained.

[0197] Further, during the L-1th round of training, obtaining the position loss value during the L-1th round of training includes: obtaining the second predicted position and the labeled position of each key point during the L-1th round of training, and obtaining the position loss value of the key point during the L-1th round of training based on the difference between the second predicted position and the labeled position of each key point. Obtaining the position loss value of the key point during the L-1th round of training based on the difference between the second predicted position and the labeled position of each key point includes: calculating the difference between the second predicted position and the labeled position of each key point during the L-1th round of training, and calculating the mean of the sum of squares between each difference, and determining it as the position loss value during the L-1th round of training based on the mean of the sum of squares.

[0198] During the L-1th round of training, the angle loss value during the L-1th round of training is obtained, including: obtaining the second predicted angle between each predicted directed line segment and the preset direction line segment during the L-1th round of training, and the second reference angle between the corresponding reference directed line segment and the preset direction line segment. Since the number of angles in each round of training is the same, the number of second predicted angles and reference angles during the L-1th round of training is also M. The difference between the M second predicted angles and the corresponding reference angles during the L-1th round of training is obtained, and the M differences are summed, and the absolute value of the accumulated value after the summation is compared with the preset difference. When the absolute value of the accumulated value is less than or equal to the preset difference, the angle loss value during the L-1th round of training is determined according to the difference between the absolute value of the accumulated value and the first preset value; when the absolute value of the accumulated value is greater than the preset difference, the angle loss value during the L-1th round of training is determined according to the product of the square of the accumulated value and the second preset value.

[0199] In this embodiment, the position loss value of the key point in the L-1 round of training is accurately calculated through the second predicted position and marked position of each key point in the L-1 round of training; the angle loss value in the L-1 round of training is accurately calculated through the second predicted angle and the corresponding reference angle in the L-1 round of training; and then the two-part loss of position and angle in the L-1 round of training is considered to accurately obtain the second weight coefficient of the second loss value in the L-1 round of training, so as to accurately control the second loss value through the second weight coefficient, thereby improving the accuracy of the training prediction model.

[0200] In one embodiment, the position loss values ​​include K; the angle loss values ​​include K; K is a positive integer; based on the position loss value and the angle loss value, obtaining a second weight coefficient of the second loss value in the L-1th round of training process includes:

[0201] Obtain a first variance of the K position loss values ​​and a second variance of the K angle loss values; and determine a second weight coefficient of the second loss value in the L-1th round of training according to a ratio of the first variance to the second variance.

[0202] Among them, variance is the degree of deviation from the data center, which is used to measure the fluctuation of a batch of data. The variance is expressed as the average of the sum of the squares of the difference between each data and the mean.

[0203] Specifically, the sample image includes K sample images, where K is a positive integer; that is, the sample image can be one sample image or multiple sample images. During the L-1 round of training, each sample image corresponds to a position loss value and an angle loss value.

[0204] Obtain the K position loss values ​​corresponding to the K sample images during the L-1th round of training, and perform variance calculation on the K position loss values ​​to obtain the first variance of the K position loss values; obtain the second variance of the K angle loss values ​​by performing variance calculation on the K angle loss values.

[0205] Furthermore, according to the ratio of the first variance to the second variance, the second weight coefficient of the second loss value in the L-1 round of training is determined. It can be seen that, on the basis that the second weight coefficient is the ratio of the first variance to the second variance, the second weight coefficient of the second loss value is related to the fluctuation of the angle loss value in the L-1 round of training to suppress the fluctuation of the angle loss. When the fluctuation of the angle loss value in the L-1 round of training is too large, the second weight coefficient will decrease accordingly. On the contrary, when the fluctuation of the angle loss value in the L-1 round of training is too small, the second weight coefficient is a larger coefficient. Therefore, the angle loss value can be maintained within a stable range to avoid interference with the model training process. In practical applications, the second weight coefficient of the second loss value in the first round of training is generally set to 0.

[0206] In one embodiment, the process of calculating the variance of K position loss values ​​includes: obtaining the average value of the K position loss values, and subtracting each position loss value from the average value of all position loss values ​​to obtain the difference of the K position loss values, calculating the sum of squares of the differences of the K position loss values, and then dividing the sum of squares by K to obtain a first variance.

[0207] In one embodiment, the process of calculating the variance of K angle loss values ​​includes: obtaining the average value of the K angle loss values, and subtracting each angle loss value from the average value of all angle loss values ​​to obtain the difference of the K angle loss values, calculating the sum of squares of the differences of the K angle loss values, and then dividing the sum of squares by K to obtain a second variance.

[0208] In an embodiment, the sample images may further include K batches of sample images. When the sample images include K batches of sample images, each batch of sample images includes the same number of sample images.

[0209] In the case where the sample images include K batches of sample images, the position loss values ​​corresponding to the K batches of sample images in the previous training process are obtained, and the first variance of the K batches of position loss values ​​is obtained by performing variance calculation on the position loss values ​​corresponding to the K batches of sample images; the second variance of the K batches of angle loss values ​​is obtained by performing variance calculation on the angle loss values ​​corresponding to the K batches of sample images.

[0210] In this embodiment, by determining the ratio of the first variance to the second variance in the L-1th round of training as the second weight coefficient of the second loss value in the Lth round of training, it is possible to suppress the fluctuation of the angle loss value when the fluctuation of the angle loss value is large or too small in the L-1th round of training, maintain the stability of the angle loss value, make the model converge normally, and improve the effect of model training.

[0211] In an exemplary embodiment, Figure 5 As shown, a posture prediction method is provided, which is applied to Figure 1 The server 104 in FIG. 1 is used as an example for explanation. Among them:

[0212] S700, predicting the position information of the key points of the target object in the image to be predicted by using the prediction model to obtain the target position information of the key points.

[0213] The image to be predicted can be a single person image or a multi-person image, but the image to be predicted needs to contain a target object. The target object can be a human object or a non-human object. The prediction model is obtained according to the training method of the prediction model.

[0214] Specifically, the server receives a posture prediction request, obtains the image to be predicted in the posture prediction request, and preprocesses the image to be estimated. The preprocessing includes: when the image to be predicted is a multi-person image, extracting a single-person image containing a target object whose posture needs to be predicted from the image to be predicted, and then performing posture prediction on the target object in the single-person image.

[0215] Furthermore, the trained prediction model supports predicting the position information of key points of the target object in a single person image. After the prediction model is trained, it can be stored in a database. When it is necessary to evaluate the image to be predicted, the prediction model can be retrieved from the database, and the position information of the key points of the target object in the image to be predicted can be predicted by the prediction model to obtain the target position information of the key points, such as: predicting the target position of the nose, predicting the target position of the left eye, predicting the right eye position of the nose, etc. In practical applications, the predicted target positions of the key points are often displayed in the form of heat maps.

[0216] Furthermore, based on the target position information of the key points, the target directed line segment formed between the key points can be determined, and then the target angle between the target directed line segment and the preset direction straight line can be determined as auxiliary reference information in the posture prediction process.

[0217] S800, determining the posture information of the target object based on the target position information.

[0218] Specifically, based on the target position information, the posture information of the target object in the image to be predicted is determined. Wherein, based on the target position information, the posture information of the target object in the image to be predicted is determined, including but not limited to: aggregating the target position information of key points, processing and analyzing the aggregated target position information of the key points, and obtaining the comprehensive posture information of the image to be predicted; or, determining the target positions of the target points as the posture information of the image to be predicted, and the posture information of the image to be predicted includes the target position information of each key point.

[0219] In the above-mentioned posture prediction method, the position information of the key points of the target object in the image to be predicted is predicted by the prediction model to obtain the target position information of the key points; based on the target position information, the posture information of the target object is identified. In the whole process, the position of the key points of the target object in the image to be predicted is accurately predicted by the prediction model that can make accurate predictions, so as to accurately identify the posture of the target object in the image to be predicted based on the target position information of the key points.

[0220] In a specific application example, taking the human posture correction scenario of fitness movements as an example, it is necessary to judge whether the human body's movements are standard and whether the force is correct based on the human body's posture in the video. At this time, a relatively stable human body posture prediction is required. The specific process of human body posture prediction is as follows:

[0221] 1. The training phase of the prediction model for human posture prediction is as follows Figure 6 shown.

[0222] S1. Input data. Collect human images of relevant scenes from existing data sources or open source data sources, and select some images from the collected images as sample images. If the sample image is a multi-person image, it is necessary to determine the target object in the multi-person image to update the sample image, and only retain the area where the target object is located in the updated sample image.

[0223] In the sample image, 17 common human key points are annotated, including: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle. The position information of the key points is marked to obtain the annotated position of the key points.

[0224] Compared with the line segments of the torso, the line segments of the limbs tend to fluctuate more. Therefore, based on the connection relationship between the key points, that is, the connection relationship between the human joints, the reference directed line segments of the limbs are determined as follows: the direction of the right upper arm is 5→7, the direction of the right forearm is 7→9, the direction of the left upper arm is 6→8, the direction of the left forearm is 8→10, the direction of the right thigh is 11→13, the direction of the right calf is 13→15, the direction of the left thigh is 12→14, and the direction of the left calf is 14→16.

[0225] Take the right upper arm 5→7 as an example. Since 5 is the starting point of the right upper arm, a horizontal right-directed line segment is set with 5 as the base point, and the reference angle between the clockwise horizontal right-directed line segment and the right upper arm is obtained. The other reference directed line segments are similarly obtained to obtain the reference angle between each target directed line segment and the horizontal line. The sample image is input into the prediction model.

[0226] S2, model training forward reasoning.

[0227] The sample image is forward inferred by the prediction model to identify the predicted position of the key point in the sample image, and the predicted directed line segment is determined based on the predicted position of the key point. Further, the predicted angle between the predicted directed line segment and the horizontal right-directed line segment is determined by using trigonometric functions.

[0228] S3. Loss calculation of key point positions.

[0229] For each sample image, the difference between the marked position of the key point in each sample image and the predicted position of the key point is calculated, and the first loss value LOSS1 is obtained based on the difference between the predicted positions of N key points in the sample image and the corresponding marked positions.

[0230] When calculating LOSS1, let the total number of key points in the sample image be N, and for any key point i in the sample image, let the estimated position of the key point be f(x i ), the sample position is y i , then the position loss LOSS1 is:

[0231]

[0232] S4. Angle loss calculation.

[0233] For each sample image, there are M predicted angles and reference angles in the sample image, and the difference between each reference angle and the corresponding predicted angle is calculated, and based on the difference between the M reference angles and the corresponding predicted angle, a second loss value LOSS2 is obtained.

[0234] When calculating LOSS2, let the prediction angle be F(l j ), the reference angle is a j, then the second difference between the predicted angle and the corresponding reference angle is F(l j )- a j , further, calculate the difference and Σ of the M second differences M (F(l j )-a j ). Let the preset difference be 1, the first preset value be 0.5, and the second preset value be 0.5. Based on the sum of the difference of the second difference, the formula for calculating the second loss value is as follows:

[0235] LOSS2=|Σ(F(l j )- a j )|-0.5,|Σ(F(l j )- a j )|>1

[0236] LOSS2=0.5(Σ(F(l j )- a j ) 2 ,|Σ(F(l j )- a j )|≤1

[0237] S5. Adaptive LOSS weight model of loss.

[0238] Based on the second loss value LOSS2, the first loss value LOSS1 is updated, and the updated first loss value is determined as the third loss value LOSS. Specifically, when the current training round is L rounds, the first weight coefficient of the second loss value in the L round training process is obtained. , and obtain the second weight coefficient of the second loss value in the L-1 round of training ; Based on the first weight coefficient With the second weight coefficient The product of LOSS1 and LOSS2 gives the angle loss weight coefficient of the second loss value during the Lth round of training; that is, LOSS = LOSS1 + * * LOSS2.

[0239] In order to avoid large fluctuations in angle loss in the early stage of model training, which makes the model unable to converge quickly, you can set =min(e / h, 1.2), where e is the number of trainings in the current training process, and h is half of the total number of preset trainings of the prediction model. It can increase with the increase of training rounds, and the maximum value does not exceed 1.2. In the later stage of model training, it gradually changes to a model training process dominated by angle loss.

[0240] Furthermore, in order to suppress the excessive fluctuation of the angle loss, it is possible to set = The first variance during the L-1th round of training / the second variance during the L-1th round of training. The first variance is obtained by calculating the variance of the K position loss values ​​during the L-1th round of training, and the second variance is obtained by calculating the variance of the K angle loss values ​​during the L-1th round of training.

[0241] S6. Model training back propagation.

[0242] After calculating the third loss value of the sample image in the Lth round of training, the model can be back-propagated based on the third loss value to update the parameters in the model, and the above training process can be repeated until the model reaches a preset number of training times or the third loss value tends to be stable, thereby obtaining a trained prediction model for human posture prediction. The trained prediction model has better robustness and stability, which is convenient for subsequent accurate evaluation of human body movements during fitness posture correction.

[0243] 2. Application phase of the model

[0244] Acquire an image to be predicted for human motion assessment; when the image to be predicted is a multi-person image, it is necessary to first extract a single person image for which posture prediction is required from the image to be predicted, and then perform human posture prediction; perform posture prediction on the target object in the image to be predicted through a prediction model, and obtain target position information of key points of the target object in the image to be predicted; determine the target position information of the key points as the human posture information of the target object, and the human posture information of the image to be predicted includes the target position information of each key point.

[0245] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0246] Based on the same inventive concept, the embodiment of the present application also provides a prediction model training device for implementing the prediction model training method involved above, and a posture prediction device for implementing the posture prediction method involved above. The implementation solutions to the problems provided by the two devices are similar to the implementation solutions recorded in the above methods, so the specific limitations in the embodiments of the training device for one or more prediction models provided below can refer to the limitations of the prediction model training method above, and the specific limitations in the embodiments of the posture prediction device provided below can refer to the limitations of the posture prediction method above, which will not be repeated here.

[0247] In an exemplary embodiment, Figure 7 As shown, a training device for a prediction model is provided, comprising: an angle acquisition module 100, a position prediction module 200, an angle prediction module 300, a first loss module 400, a second loss module 500 and a training module 600, wherein:

[0248] An angle acquisition module 100 is used to identify a reference directed line segment formed between key points of a target object in a sample image, determine an angle between the reference directed line segment and a preset direction line segment, and obtain a reference angle;

[0249] The position prediction module 200 is used to predict the sample image and obtain the first predicted position of the key point of the target object;

[0250] An angle prediction module 300, used to determine a predicted directed line segment according to a first predicted position of the key point, and determine a first predicted angle according to an angle between the predicted directed line segment and a preset direction line segment;

[0251] A first loss module 400, configured to determine a first loss value according to the first predicted position and the marked position of the key point of the target object;

[0252] A second loss module 500, configured to determine a second loss value according to a reference angle and a first predicted angle;

[0253] The training module 600 is used to train the prediction model according to the first loss value and the second loss value.

[0254] In one embodiment, the angle acquisition module 100 is further used to acquire candidate directed line segments formed by key points; based on the connection relationship between the key points, a reference directed line segment is obtained from the candidate directed line segments.

[0255] In one embodiment, the angle acquisition module 100 is also used to identify the starting key point of the reference directed line segment; obtain a preset direction line segment starting from the starting key point; determine the angle between the reference directed line segment and the preset direction line segment to obtain a reference angle.

[0256] In one embodiment, the key points include N; N is an integer greater than or equal to 2; each key point includes a first predicted position and a marked position; the first loss module 400 is also used to determine the first difference between the first predicted position and the marked position of each key point, and obtain N first differences; based on the mean of the sum of the squares of the N first differences, determine the first loss value.

[0257] In one embodiment, the reference angles include M; the first prediction angles include M; the first prediction angles correspond one-to-one to the reference angles; M is a positive integer; the second loss module 500 is also used to determine, for each of the M first prediction angles, the second difference between the first prediction angle and the corresponding reference angle to obtain M second differences; the M second differences are summed to obtain a difference sum; when the absolute value of the difference sum is less than or equal to the preset difference, the second loss value is determined based on the difference between the absolute value of the difference sum and the first preset value; when the absolute value of the difference sum is greater than the preset difference, the second loss value is determined based on the product of the square of the difference sum and the second preset value.

[0258] In one embodiment, the training module 600 is also used to obtain the angle loss weight coefficient of the second loss value during the Lth round of training when the current training round is L rounds; wherein L is an integer greater than 1 and less than or equal to C; C presets the total number of training times; based on the angle loss weight coefficient, the first loss value and the second loss value are weighted and summed to obtain a third loss value; based on the third loss value, the prediction model is trained.

[0259] In one embodiment, the training module 600 is also used to obtain a first weight coefficient of the second loss value during the Lth round of training, and to obtain a second weight coefficient of the second loss value during the L-1th round of training; based on the product of the first weight coefficient and the second weight coefficient, the angle loss weight coefficient of the second loss value during the Lth round of training is obtained.

[0260] In one embodiment, the training module 600 is also used to determine the ratio between the number of training times in the Lth round of training and half of the total number of training times; when the ratio is less than a preset threshold, the ratio is determined as the first weight coefficient of the second loss value in the Lth round of training; when the ratio is greater than or equal to the preset threshold, the preset threshold is determined as the first weight coefficient of the second loss value in the Lth round of training.

[0261] In one embodiment, the training module 600 is further used to obtain the second predicted position and the marked position of each key point in the L-1 round of training, and determine the position loss value of the key point in the L-1 round of training based on the second predicted position and the marked position;

[0262] Obtain the second predicted angle between each predicted directed line segment and the preset direction line segment in the L-1th round of training, and the reference angle between the corresponding reference directed line segment and the preset direction line segment, wherein the reference angles include M; the predicted directed line segments correspond one-to-one to the reference directed line segments; the second predicted angles correspond one-to-one to the reference angles; based on the M second predicted angles and the corresponding reference angles, determine the angle loss value in the L-1th round of training; based on the position loss value and the angle loss value, obtain the second weight coefficient of the second loss value in the L-1th round of training.

[0263] In one embodiment, the position loss values ​​include K; the angle loss values ​​include K; K is a positive integer; the training module 600 is also used to obtain the first variance of the K position loss values ​​and the second variance of the K angle loss values; according to the ratio of the first variance to the second variance, determine the second weight coefficient of the second loss value in the L-1th round of training.

[0264] In an exemplary embodiment, Figure 8 As shown, a posture prediction device is provided, including: a posture prediction module 700 and a posture recognition module 800, wherein:

[0265] The posture prediction module 700 is used to predict the position information of the key points of the target object in the image to be predicted by using the prediction model to obtain the target position information of the key points;

[0266] The posture recognition module 800 is used to recognize the posture information of the target object based on the target position information; wherein the prediction model is obtained according to the training method of the prediction model.

[0267] The above-mentioned human posture estimation model training device and each module in the human posture estimation device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0268] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a human body posture estimation model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a human body posture estimation model training method and a human body posture estimation method are implemented.

[0269] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0270] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0271] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0272] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0273] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0274] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0275] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A prediction model training method, characterized in that: include: Identify a reference directed line segment formed by key points of a target object in a sample image, and determine an angle between the reference directed line segment and a preset direction line segment to obtain a reference angle; Predicting the sample image to obtain a first predicted position of a key point of the target object; Determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to an angle between the predicted directed line segment and a preset direction line segment; Determining a first loss value according to the first predicted position and the marked position of the key point of the target object; Determining a second loss value according to the reference angle and the first predicted angle; The prediction model is trained according to the first loss value and the second loss value.

2. The method according to claim 1, characterized in that The step of identifying a reference directed line segment formed by key points of a target object in a sample image includes: Obtain candidate directed line segments formed by the key points; Based on the connection relationship between the key points, a reference directed line segment is obtained by screening from the candidate directed line segments.

3. The method according to claim 1, characterized in that The step of determining the angle between the reference directed line segment and the preset direction line segment to obtain a reference angle includes: Identifying a starting key point of the reference directed line segment; Obtaining a preset direction line segment starting from the starting key point; The angle between the reference directed line segment and the preset direction line segment is determined to obtain a reference angle.

4. The method according to claim 1, characterized in that: The key points include N; N is an integer greater than or equal to 2; each key point includes the first predicted position and the marked position; The determining a first loss value according to the first predicted position and the marked position of the key point of the target object includes: Determine a first difference between the first predicted position and the marked position of each of the key points to obtain N first differences; A first loss value is determined based on an average of the sums of squares of the N first differences.

5. The method according to claim 1, characterized in that The reference angles include M; the first prediction angles include M; the first prediction angles correspond one-to-one to the reference angles; M is a positive integer; The determining a second loss value according to the reference angle and the first predicted angle includes: For each first prediction angle among the M first prediction angles, determine a second difference between the first prediction angle and the corresponding reference angle to obtain M second difference values; Performing sum processing on the M second differences to obtain a difference sum; When the absolute value of the sum of the differences is less than or equal to the preset difference, determining a second loss value according to the difference between the absolute value of the sum of the differences and the first preset value; When the absolute value of the sum of the differences is greater than the preset difference, the second loss value is determined according to the product of the square of the sum of the differences and a second preset value.

6. The method according to claim 5, characterized in that The step of training the prediction model according to the first loss value and the second loss value includes: When the current training round is L rounds, obtain the angle loss weight coefficient of the second loss value in the L round training process; wherein L is an integer greater than 1 and less than or equal to C; C presets the total number of training times; Based on the angle loss weight coefficient, weighted summing the first loss value and the second loss value is performed to obtain a third loss value; Based on the third loss value, the prediction model is trained.

7. The method according to claim 6, characterized in that The step of obtaining the angle loss weight coefficient of the second loss value during the L-th round of training includes: Obtaining a first weight coefficient of the second loss value during the Lth round of training, and obtaining a second weight coefficient of the second loss value during the L-1th round of training; Based on the product of the first weight coefficient and the second weight coefficient, an angle loss weight coefficient of the second loss value in the Lth round of training is obtained.

8. The method according to claim 7, characterized in that The obtaining of the first weight coefficient of the second loss value in the L-th round of training includes: Determine the ratio between the number of training times of the Lth round of training and half of the total number of training times; When the ratio is less than a preset threshold, determining the ratio as a first weight coefficient of the second loss value in the Lth round of training; When the ratio is greater than or equal to a preset threshold, the preset threshold is determined as a first weight coefficient of the second loss value in the Lth round of training.

9. The method according to claim 7, characterized in that: The obtaining of the second weight coefficient of the second loss value in the L-1th round of training includes: Obtaining a second predicted position of each of the key points during the L-1th round of training, and determining a position loss value of the key point during the L-1th round of training based on the second predicted position and the marked position; Acquire a second predicted angle between each of the predicted directed line segments and the preset directional line segment during the L-1th round of training, wherein the predicted directed line segment corresponds to the reference directed line segment in one-to-one correspondence; and the second predicted angle corresponds to the reference angle in one-to-one correspondence; Determine an angle loss value in an L-1th round of training based on the M second predicted angles and the corresponding reference angles; Based on the position loss value and the angle loss value, a second weight coefficient of the second loss value in the L-1th round of training is obtained.

10. The method according to claim 9, characterized in that The position loss values ​​include K; the angle loss values ​​include K; K is a positive integer; and obtaining a second weight coefficient of the second loss value in the L-1 round of training based on the position loss value and the angle loss value includes: Obtaining first variances of the K position loss values ​​and second variances of the K angle loss values; A second weight coefficient of the second loss value during the L-1th round of training is determined according to a ratio of the first variance to the second variance.

11. A posture prediction method, characterized in that: include: Predicting the position information of the key points of the target object in the image to be predicted by the prediction model to obtain the target position information of the key points; Based on the target position information, identifying the posture information of the target object; Wherein, the prediction model is trained according to the training method according to any one of claims 1-10.

12. A prediction model training device, characterized in that: include: An angle acquisition module, used to identify the reference directed line segment formed between the key points of the target object in the sample image, determine the angle between the reference directed line segment and the preset direction line segment, and obtain a reference angle; A position prediction module, used to predict the sample image to obtain a first predicted position of a key point of the target object; An angle prediction module, used to determine a predicted directed line segment according to the first predicted position of the key point, and determine a first predicted angle according to an angle between the predicted directed line segment and a preset direction line segment; A first loss module, configured to determine a first loss value according to the first predicted position and the marked position of the key point of the target object; A second loss module, configured to determine a second loss value according to the reference angle and the first predicted angle; A training module is used to train the prediction model according to the first loss value and the second loss value.

13. A posture prediction device, characterized in that: include: A posture prediction module, used to predict the position information of the key points of the target object in the image to be predicted through a prediction model, and obtain the target position information of the key points; A posture recognition module is used to identify the posture information of the target object based on the target position information; wherein the prediction model is obtained according to the training method according to any one of claims 1-10.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images