Three-dimensional posture determination method, device, electronic device and storage medium

Abstract

The present disclosure relates to a method, device, electronic device and storage medium for determining a three-dimensional posture, the method comprising: identifying an image containing an object in a video frame to be processed; determining the confidence of a key point of a part included in the image, and obtaining a first parameter value of a three-dimensional posture model of the object; processing the posture information of the object according to the confidence of the key point of the part to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value; summing up each first posture constraint item to determine a posture optimization item; determining a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object. The inclusion relationship between the key points of the part of the object and the image is taken into consideration, that is, whether the joints are out of the picture is taken into consideration, thereby improving the motion capture effect and determining a more accurate three-dimensional posture.

Description

Technical Field

[0001] The present application relates to the field of motion capture technology, and in particular to a method, device, electronic device and storage medium for determining a three-dimensional posture. Background Art

[0002] Motion capture refers to capturing the positions of key parts of a moving object to determine the movement of the moving object. After capturing the movement of a moving object, it can be applied in animation production, gait analysis, biomechanics or ergonomics. For example, in the fields of live entertainment, human-computer interaction and virtual content production, motion capture can be used to drive virtual characters for live broadcasts. After capturing the movement of a moving object, the three-dimensional posture of the moving object can be determined based on the captured movement.

[0003] In the related art, the commonly used motion capture method is to use a single color camera for motion capture. However, this method is mainly aimed at full-body scenes, which requires the user to be far away from the camera, which is not convenient to use. In the fields of short videos or live broadcasts, users usually only have their upper body in the picture, and as the camera posture or user movements change, the upper body joints will frequently enter and exit the picture, resulting in poor capture effect. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device and storage medium for determining a three-dimensional posture, so as to improve the motion capture effect and thereby improve the accuracy of three-dimensional posture determination.

[0005] In a first aspect, an embodiment of the present application provides a method for determining a three-dimensional posture, including:

[0006] Identifying an image containing an object in a video frame to be processed;

[0007] Determine the confidence of the key points of the parts included in the image, and obtain a first parameter value of the three-dimensional posture model of the object;

[0008] Processing the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value;

[0009] Adding up the first posture constraint items to determine a posture optimization item;

[0010] Determine a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

[0011] In some exemplary embodiments, the confidence level of each part key point is determined by: determining a two-dimensional heat map of the part key points included in the image;

[0012] For each part key point, the confidence of the part key point is determined according to the resolution of the two-dimensional heat map and the grayscale value of the pixel, wherein the confidence represents the probability that the image contains the corresponding part key point.

[0013] In some exemplary embodiments, determining the confidence of the key point of the part according to the resolution of the two-dimensional heat map and the grayscale value of the pixel point includes:

[0014] Determining pixel features of the two-dimensional heat map according to grayscale values ​​of pixels of the two-dimensional heat map and a resolution of the two-dimensional heat map;

[0015] The pixel features are transformed using a set probability algorithm to obtain the confidence of the key points of the part.

[0016] In some exemplary embodiments, the method further comprises:

[0017] Determine each second posture constraint item according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value;

[0018] Add each of the second posture constraint items and the posture optimization item to determine an updated posture optimization item;

[0019] Determine a second parameter value of the three-dimensional posture model of the object when the updated posture optimization item takes a minimum value.

[0020] In some exemplary embodiments, the first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item, and a bone orientation constraint item; the second posture constraint item includes at least one of a posture prior constraint item and a posture regularization constraint item.

[0021] In some exemplary embodiments, the two-dimensional projection constraint is determined in the following manner:

[0022] The two-dimensional projection constraint item is determined based on the preset projection constraint item weight, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each of the part key points.

[0023] In some exemplary embodiments, the two-dimensional projection constraint is determined using the following formula:

[0024]

[0025] Among them, E 2d represents the two-dimensional projection constraint term, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of the part key points, K represents the projection matrix, P n (θ, β) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

[0026] In some exemplary embodiments, the three-dimensional constraint term is determined as follows:

[0027] The three-dimensional constraint item is determined based on the preset weight of the three-dimensional part key point position constraint item, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the three-dimensional position information of each of the part key points.

[0028] In some exemplary embodiments, the three-dimensional constraint term is determined using the following formula:

[0029]

[0030] Among them, E 3d represents the three-dimensional projection constraint term, ω 3d represents the weight of the preset three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of the part key points, P n (θ, β) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the three-dimensional position information of the key point of the nth part obtained.

[0031] In some exemplary embodiments, the bone orientation constraint is determined as follows:

[0032] The bone orientation constraint item is determined according to a preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

[0033] In some exemplary embodiments, the following formula is used to determine the bone orientation constraint:

[0034]

[0035] Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, D r (θ, β) represents the orientation of each bone determined by the object model parameters, d r is the orientation of the rth bone determined by the two key points of the skeleton.

[0036] In some exemplary embodiments, the confidence of each of the bones is determined by:

[0037] The confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the probability that the skeleton contains the valid part key point; the valid part key point is a part key point whose confidence is greater than a preset confidence threshold.

[0038] In some exemplary embodiments, the orientation of each of the bones is determined as follows:

[0039] The orientation of the skeleton is determined based on the number of valid part key points included in the two part key points constituting the skeleton.

[0040] In some exemplary embodiments, the posture prior constraint is determined in the following manner:

[0041] The posture prior constraint item is determined according to the preset posture prior constraint weight, the preset covariance matrix, the preset bone rotation parameter mean and the bone rotation angle parameter in the object model parameter.

[0042] In some exemplary embodiments, the posture prior constraint term is determined using the following formula:

[0043] E prior =ω prior ‖A1(θ-b1)‖ 2

[0044] Among them, E prior represents the posture prior constraint, ω priorrepresents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

[0045] In some exemplary embodiments, the posture regularization constraint is determined in the following manner:

[0046] The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

[0047] In some exemplary embodiments, the posture regularization constraint term is determined using the following formula:

[0048] E reg =ω pose，reg ‖θ-θ init ‖ 2 +ω shape，reg ‖β-β ibit ‖ 2

[0049] Among them, E reg represents the attitude regularization constraint, ω pose，reg represents the pre-set posture regression weight, ω shape，reg represents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, θ init represents the initial rotation angle value of the skeleton of the object, β represents the object body parameter in the object model parameter, β init An initial coefficient value representing the object's body shape.

[0050] In some exemplary embodiments, determining a two-dimensional heat map of key points of a part included in the image includes:

[0051] An image containing an object in the video to be processed is used as an input of a two-dimensional part key point network model to obtain a two-dimensional heat map of the part key points.

[0052] In some exemplary embodiments, the three-dimensional position information of the key points of the part is determined by:

[0053] Using the image containing the object in the video frame to be processed as the input of the three-dimensional part key point network model to obtain a three-dimensional heat map of the part key points;

[0054] According to the three-dimensional heat map, the three-dimensional position information of the key points of the part is determined.

[0055] In a second aspect, an embodiment of the present application provides a three-dimensional posture determination device, including:

[0056] An image recognition unit is configured to recognize an image containing an object in a video frame to be processed;

[0057] A confidence determination unit, configured to determine the confidence of the key points of the parts included in the image, and obtain a first parameter value of the three-dimensional posture model of the object;

[0058] A basic constraint item determination unit is configured to process the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value;

[0059] a posture optimization item determination unit, configured to perform summing of each of the first posture constraint items to determine a posture optimization item;

[0060] The posture determination unit is configured to determine a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

[0061] In some exemplary embodiments, the confidence determination unit is specifically configured to execute:

[0062] Determine a two-dimensional heat map of key points of the part included in the image;

[0063] For each part key point, the confidence of the part key point is determined according to the resolution of the two-dimensional heat map and the grayscale value of the pixel, wherein the confidence represents the probability that the image contains the corresponding part key point.

[0064] In some exemplary embodiments, the confidence determination unit is configured to perform:

[0065] Determining pixel features of the two-dimensional heat map according to grayscale values ​​of pixels of the two-dimensional heat map and a resolution of the two-dimensional heat map;

[0066] The pixel features are transformed using a set probability algorithm to obtain the confidence of the key points of the part.

[0067] In some exemplary embodiments, the method further includes an updating unit configured to execute:

[0068] Determine each second posture constraint item according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value;

[0069] Add each of the second posture constraint items and the posture optimization item to determine an updated posture optimization item;

[0070] Determine a second parameter value of the target model of the three-dimensional posture of the object when the updated posture optimization item takes a minimum value.

[0071] In some exemplary embodiments, the first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item, and a bone orientation constraint item; the second posture constraint item includes at least one of a posture prior constraint item and a posture regularization constraint item.

[0072] In some exemplary embodiments, the posture determination unit is configured to determine the two-dimensional projection constraint item in the following manner:

[0073] The two-dimensional projection constraint item is determined based on the preset projection constraint item weight, the probability parameter of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each of the part key points.

[0074] In some exemplary embodiments, the posture determination unit is configured to determine the two-dimensional projection constraint item by applying the following method:

[0075]

[0076] Among them, E 2d represents the two-dimensional projection constraint term, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of the part key points, K represents the projection matrix, P n (θ, β)) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

[0077] In some exemplary embodiments, the posture determination unit is configured to determine the three-dimensional constraint item in the following manner:

[0078] The three-dimensional constraint item is determined based on the preset weight of the three-dimensional part key point position constraint item, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the three-dimensional position information of each of the part key points.

[0079] In some exemplary embodiments, the posture determination unit is configured to perform the following formula to determine the three-dimensional constraint item:

[0080]

[0081] Among them, E 3d represents the three-dimensional projection constraint term, ω 3d represents the weight of the preset three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of the part key points, P n (θ, β)) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the three-dimensional position information of the key point of the nth part obtained.

[0082] In some exemplary embodiments, the posture determination unit is configured to determine the skeleton orientation constraint item in the following manner:

[0083] The bone orientation constraint item is determined according to a preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

[0084] In some exemplary embodiments, the posture determination unit is configured to perform the following formula to determine the skeleton orientation constraint:

[0085]

[0086] Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, D r (θ, β) represents the orientation of each bone determined by the object model parameters, d r is the orientation of the rth bone determined by the two key points of the skeleton.

[0087] In some exemplary embodiments, the posture determination unit is configured to determine the confidence of each of the bones in the following manner:

[0088] The confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the confidence that the skeleton contains the valid part key points; the valid part key points are part key points whose confidence is greater than a preset confidence threshold.

[0089] In some exemplary embodiments, the posture determination unit is configured to determine the orientation of each of the bones in the following manner:

[0090] The orientation of the skeleton is determined based on the number of valid part key points included in the two part key points constituting the skeleton.

[0091] In some exemplary embodiments, the posture determination unit is configured to determine the posture prior constraint item in the following manner:

[0092] The posture prior constraint item is determined according to the preset posture prior constraint weight, the preset covariance matrix, the preset bone rotation parameter mean and the bone rotation angle parameter in the object model parameter.

[0093] In some exemplary embodiments, the posture determination unit is configured to apply the following formula to determine the posture prior constraint item:

[0094] E prior =ω prior ‖A1(θ-b1)‖ 2

[0095] Among them, E prior represents the posture prior constraint, ω prior represents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

[0096] In some exemplary embodiments, the posture determination unit is configured to determine the posture regularization constraint item in the following manner:

[0097] The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

[0098] In some exemplary embodiments, the posture determination unit is configured to perform the following formula to determine the posture regularization constraint:

[0099] E reg =ω pose，reg ‖θ-θ init ‖ 2 +ω shape，reg ‖β-β init ‖ 2

[0100] Among them, E reg represents the attitude regularization constraint, ω pose，reg represents the pre-set posture regression weight, ω shape，reg represents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, θ init represents the initial rotation angle value of the skeleton of the object, β represents the object body parameter in the object model parameter, β init An initial coefficient value representing the object's body shape.

[0101] In some exemplary embodiments, the confidence determination unit is configured to perform:

[0102] An image containing an object in the video to be processed is used as an input of a two-dimensional part key point network model to obtain a two-dimensional heat map of the part key points.

[0103] In some exemplary embodiments, a three-dimensional position information determining unit is further included, and the three-dimensional position information determining unit is configured to determine the three-dimensional position information of the key point of the part in the following manner:

[0104] Using the image containing the object in the video frame to be processed as the input of the three-dimensional part key point network model to obtain a three-dimensional heat map of the part key points;

[0105] According to the three-dimensional heat map, the three-dimensional position information of the key points of the part is determined.

[0106] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0107] processor;

[0108] a memory for storing instructions executable by the processor;

[0109] The processor is configured to execute the instructions to implement any one of the above three-dimensional posture determination methods.

[0110] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any one of the above-mentioned three-dimensional posture determination methods.

[0111] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any one of the above-mentioned three-dimensional posture determination methods when executed by a processor.

[0112] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0113] By identifying the image containing the object in the video frame to be processed, the confidence of each part key point included in the image can be obtained, and the confidence represents the probability that the image contains the corresponding part key point; in this way, the inclusion relationship between each part key point and the image can be determined. Taking live broadcast as an example, if the image contains the part key point, it indicates that the part key point is not out of the picture, otherwise it indicates that the part key point is out of the picture. Therefore, when determining the three-dimensional posture of the object, the posture information of the object is processed according to the confidence of the part key point to determine each first posture constraint item, and each first posture constraint item is summed to determine the posture optimization item; and then the second parameter value of the three-dimensional posture model of the object when the posture optimization item takes the minimum value is used as the parameter representing the three-dimensional posture of the object. In this way, when the posture optimization item is reversed, the inclusion relationship between the key point representing the part and the image is taken into account. In this way, compared with the prior art, the problem of frequent out-of-picture of the upper body joints in fields such as live broadcast can be solved, the effect of motion capture is improved, and the three-dimensional posture of the object finally determined is more accurate.

[0114] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0116] Figure 1 A display interface diagram of a live broadcast process provided by an embodiment of the present application;

[0117] Figure 2 Another display interface diagram of a live broadcast process is provided for an embodiment of the present application;

[0118] Figure 3 A schematic diagram of an application scenario for determining a three-dimensional posture is provided for an embodiment of the present application;

[0119] Figure 4 A flowchart of a method for determining a three-dimensional posture is provided for an embodiment of the present application;

[0120] Figure 5 A schematic diagram of a heat map of a bone orientation is provided for an embodiment of the present application;

[0121] Figure 6 A schematic diagram of the structure of a three-dimensional posture determination device provided in one embodiment of the present application;

[0122] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0123] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0124] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0125] Below, some terms in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0126] (1) In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0127] (2) "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0128] (3) Servers provide services for terminals. The services provided include providing resources to terminals and storing terminal data. Servers correspond to applications installed on terminals and work in conjunction with applications on terminals.

[0129] (4) Terminal device, which can refer to software APP (Application) or client. It has a visual display interface and can interact with users; it corresponds to the server and provides local services to customers. For software applications, except for some applications that only run locally, they are generally installed on ordinary client terminals and need to cooperate with the server to run. After the development of the Internet, more commonly used applications include email clients for sending and receiving emails, and instant messaging clients. For this type of application, there needs to be corresponding servers and service programs in the network to provide corresponding services, such as database services, configuration parameter services, etc. In this way, specific communication connections need to be established between the client terminal and the server to ensure the normal operation of the application.

[0130] (5) Model parameters. Taking humans as an example, the model parameters are human body model parameters. In this application, the SMPL (Skinned Multi-Person Linear) model can be used. It is a parametric human body model. This method can perform arbitrary human body modeling and animation driving. The proposed method of human body posture image surface morphology can simulate the protrusions and depressions of human muscles during limb movement. Therefore, the surface distortion of the human body during movement can be avoided, and the morphology of human muscle stretching and contraction movements can be accurately portrayed. The model contains the rotation of 24 human bones and a human body shape coefficient of length 10. The bone rotation is represented as a matrix The mth row θ m represents the rotation of the mth bone (in the form of axis angle), and the human body shape coefficient is expressed as Since the upper body is usually drawn in the application scenario of the embodiment of the present application, the 12 human bones of the upper body are considered (which bones can be retained according to actual needs), and the matrix express.

[0131] (6) Part key points: Taking humans as an example, the part key points are the part key points of each part of the upper body, such as hips, shoulders, elbows, wrists, eyes, nose and other common part key points of the upper body.

[0132] (7) Out of frame: Taking live broadcast as an example, a key point of the host’s body (such as a hand) leaves the screen and is no longer in the picture.

[0133] (8) Entering the frame: Taking live broadcast as an example, a key point of the host’s body (such as a hand) enters the frame and is within the frame.

[0134] In actual practice, taking short videos or live broadcasts as an example, at least one camera is usually set up in the live broadcast room. The host usually has his upper body in the frame, and as the camera posture or user movements change, the upper body joints will frequently enter and exit the frame, resulting in poor motion capture effects. In a specific example, Figure 1 A display interface diagram of a live broadcast process is shown. Figure 2 Another display interface diagram of the live broadcast process is shown, in which: Figure 1 In the picture, the host's upper body is in the picture, and 11 is the hand; when the host moves, it may cause Figure 2 For example, the host's right hand is out of the picture, and 12 is the right hand that is out of the picture and cannot be displayed on the screen. If the right hand posture information is used to determine the host's three-dimensional posture at this time, the effect of motion capture is poor, and the accuracy of the determined three-dimensional posture of the host is low.

[0135] To this end, the present application provides a three-dimensional posture determination method, which includes identifying an image containing an object in a video frame to be processed; determining the confidence of a key point of a part included in the image, and obtaining a first parameter value of a three-dimensional posture model of the object; processing the posture information of the object according to the confidence of the key point of the part to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value; summing up each first posture constraint item to determine a posture optimization item; determining a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object. The motion capture effect is improved, thereby improving the accuracy of determining the three-dimensional posture of the object.

[0136] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0137] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In specific implementation, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0138] refer to Figure 3, which is a schematic diagram of an application scenario of the three-dimensional posture determination method provided in an embodiment of the present application. The application scenario includes multiple terminal devices 301 (including terminal device 301-1, terminal device 301-2, ... terminal device 301-n), and a three-dimensional posture determination server 302. Among them, the terminal device 301 and the three-dimensional posture determination server 302 are connected via a wireless or wired network, and the terminal device 301 includes but is not limited to electronic devices such as desktop computers, mobile phones, mobile computers, tablet computers, media players, smart wearable devices, and smart TVs. The three-dimensional posture determination server 302 can be a server cluster composed of one server, a server cluster composed of several servers, or a cloud computing center. The three-dimensional posture determination server 302 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0139] Different anchors send the video frames to be processed to the server through their respective corresponding terminal devices 301. The terminal device 301-1 corresponding to anchor 1 sends the video frames to be processed to the server 302 as an example for explanation. The video frame to be processed is a video frame at the current moment during the anchor's live broadcast. The server 302 receives the image of the object included in the video frame. The object is the anchor in this example. In actual application, the object can also be a cat or a dog. The server 302 identifies the image of the anchor contained in the video frame to be processed, obtains a two-dimensional heat map of the key points of the parts (such as arms, eyes, nose and ears, etc.), and then determines the confidence corresponding to the key point of the part according to the resolution of the two-dimensional heat map and the gray value of the pixel point for each key point of the part. The confidence represents the probability that the image contains the corresponding key point of the part, that is, some key points of the part are not in the image, and some key points of the part are in the image. In this way, when determining the three-dimensional posture of the anchor, the posture information of the object is processed according to the confidence of the key points of the part to determine each first posture constraint item, and each first posture constraint item is summed to determine the posture optimization item; finally, the posture optimization item is reversed to determine the second parameter value of the three-dimensional posture model of the object when the posture optimization item takes the minimum value. Finally, the determined second parameter value is used as the final three-dimensional posture parameter of the anchor when the anchor is showing, and is displayed through the terminal device 301-1.

[0140] Among them, the terminal device 301-1 can be a camera of a participating anchor, and the camera can execute the above-mentioned three-dimensional posture determination method without sending the video frame to be processed to the server for processing.

[0141] Of course, the method provided in the embodiment of the present application is not limited to Figure 3 The application scenarios shown can also be used in other possible application scenarios, and the embodiments of the present application are not limited thereto. Figure 3 The functions that can be implemented by each device in the application scenario shown will be described in the subsequent method embodiments, and will not be described in detail here.

[0142] To further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation steps shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application.

[0143] Combine the following Figure 4 The application scenario shown illustrates the technical solution provided by the embodiment of the present application.

[0144] refer to Figure 4 The present invention provides a method for determining a three-dimensional posture, comprising the following steps:

[0145] S401: Identify an image containing an object in a video frame to be processed.

[0146] S402: Determine the confidence of the key points of the parts included in the image, and obtain the first parameter value of the three-dimensional posture model of the object.

[0147] S403 . Processing the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value.

[0148] S404: sum up the first posture constraint items to determine the posture optimization item.

[0149] S405 , determining a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

[0150] In the embodiment of the present application, by identifying the image containing the object in the video frame to be processed, the confidence of each part key point included in the image can be obtained, and the confidence represents the probability that the image contains the corresponding part key point; in this way, the inclusion relationship between each part key point and the image can be determined. Taking live broadcast as an example, if the image contains the part key point, it indicates that the part key point is not out of the picture, otherwise it indicates that the part key point is out of the picture. Therefore, when determining the three-dimensional posture of the object, the posture information of the object is processed according to the confidence of the part key point to determine each first posture constraint item, and each first posture constraint item is summed to determine the posture optimization item; and then the second parameter value of the three-dimensional posture model of the object when the posture optimization item takes the minimum value is used as the parameter representing the three-dimensional posture of the object. In this way, when the posture optimization item is reversed, the inclusion relationship between the characterizing part key point and the image is taken into account. In this way, compared with the prior art, the problem of frequent out-of-picture of the upper body joints in fields such as live broadcast can be solved, the effect of motion capture is improved, and the three-dimensional posture of the object finally determined is more accurate.

[0151] In relation to S401, the video to be processed can be a color video shot by a camera, can be pre-shot, such as a short video, or can be a real-time video stream during a live broadcast. For the image corresponding to the video frame at the current moment in the video to be processed, the image is cropped, and the parts irrelevant to the object are removed to obtain an image including the object. The object is taken as a person (anchor or person appearing in a short video) to illustrate the cropping process. For example, a cropping frame is used to obtain the position of a human body in the image, and the image is cropped. The cropping frame can be obtained using a human body detection algorithm, such as Faster-RCNN (Faster-Region-Convolutional Neural Networks), or it can be obtained using the position of the key points of the parts extracted in the previous frame, which will not be described here.

[0152] In identifying an image containing an object in a video frame to be processed, a two-dimensional heat map of the key points of the parts is obtained. Specifically, the image containing the object in the video to be processed is used as an input of a two-dimensional network model of the key points of the parts to obtain the two-dimensional heat map of the key points of the parts. The two-dimensional grid model of the key points of the parts can be pre-trained, and the training process is not repeated.

[0153] In step S402, the confidence of the key points of the parts included in the image is determined, and a first parameter value of the three-dimensional posture model of the object is obtained.

[0154] Specifically, since a two-dimensional heat map can be obtained for each part key point, the confidence corresponding to the part key point is determined according to the resolution of the corresponding two-dimensional heat map and the grayscale value of the pixel point for each part key point. The confidence represents the probability that the image contains the corresponding part key point.

[0155] In this way, since the confidence represents the probability that the image contains the key points of the parts, it is possible to determine which key points of the parts are in the image, so as to robustly handle the situation where some limbs are not in the picture during upper body motion capture.

[0156] In a specific example, suppose H n is a two-dimensional heat map of the size of the nth part key point W*H, where H is the height of the two-dimensional heat map or the number of columns, and W is the width or the number of rows of the two-dimensional heat map. The units of height and width can be expressed in pixels. First, the two-dimensional heat map is normalized, for example, by softmax operation, to obtain the probability map of each part key point at each pixel.

[0157]

[0158] Among them, i represents the row number of the pixel, j represents the column number of the pixel, [H n ] i，j Represents the grayscale value of the pixel in the i-th row and j-th column.

[0159] After weighted averaging the above probability maps, the normalized two-dimensional coordinates of the key points of the parts can be obtained.

[0160]

[0161]

[0162] In addition, the confidence of the part key point can be determined according to the grayscale value of the pixel point of the two-dimensional heat map of the part key point. Specifically, the pixel features of the two-dimensional heat map are determined according to the grayscale value of the pixel point of the two-dimensional heat map and the resolution of the two-dimensional heat map; the pixel features are transformed using a set probability algorithm to obtain the confidence corresponding to the part key point.

[0163] In a specific example, the height H and width W of the two-dimensional heat map represent its resolution, and the confidence of the key points of the part can be determined as follows:

[0164]

[0165] Among them, c n is the confidence of the key point n, is the pixel feature in this example, which is the average value of the pixels at each pixel point, and sigmoid is the function applied in the set probability algorithm. By applying this function, the confidence obtained is 0 or 1. Since each key point of a part corresponds to a two-dimensional heat map, and the two-dimensional heat map is obtained by taking the image containing the object in the video to be processed as the input of the two-dimensional key point network model.

[0166] The confidence level determined by the above method can more accurately determine which key points of the parts are in the image.

[0167] Since the inclusion relationship between the key points of the parts and the image is determined through a two-dimensional heat map, and the two-dimensional heat map is obtained through a two-dimensional key point network model, the two-dimensional position of the key points obtained is more accurate, and the three-dimensional posture is determined more accurately.

[0168] In addition, during the training process of the two-dimensional part key point network model, the position of the two-dimensional part key point is supervised by the MSE (Mean Squared Error) loss function, and the confidence is supervised by the BCE (Binary Classification Loss) loss function, that is, when the marked part key point is inside and outside the image, the confidence supervision is 1 and 0 respectively. Therefore, the confidence of the determined part key point represents the confidence that the image contains the corresponding part key point, that is, if the confidence is 1, it means that the part key point is in the image, that is, in the short video or live broadcast, it is defined as a valid part key point. If the confidence is 0, it means that the part key point is not in the image, that is, out of the picture in the short video or live broadcast.

[0169] In addition, in order to determine the final three-dimensional posture of the object, that is, to determine the values ​​of the relevant parameters that characterize its three-dimensional posture, it is necessary to obtain the first parameter value of the first parameter of these relevant parameters of the three-dimensional posture model of the object. In a specific example, the first parameter value can be, for example, a set of values ​​of the upper body bone rotation θ, the human body shape coefficient β, and the position t of the key point of the set part, and this set of parameter values ​​can be randomly set. Since the subsequent process is to inversely optimize the posture constraint items it constitutes, its value will affect the optimization result, and the specific situation can be set according to actual needs.

[0170] In S403, after the confidence of the part key points included in the image is determined, it is possible to know which part key points are in the image and which are not in the image. In this way, when determining the three-dimensional posture of the object, on the one hand, it is determined which part key points are available and which are not available according to the confidence; on the other hand, it is determined which parameters of the part key points are applied, such as position parameters, to determine the three-dimensional posture of the object. Specifically, the posture information of the object can be processed according to the confidence of the part key points to determine each first posture constraint item, wherein the first posture constraint item represents the basic posture constraint of the first parameter value.

[0171] Optionally, the first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item and a bone orientation constraint item.

[0172] In the actual application process, the posture information can be, for example, the position parameters of the key points of each part. Exemplarily, the position parameters of the key points of the part can include the two-dimensional coordinates, three-dimensional coordinates and other parameters determined according to the two-dimensional coordinates and / or three-dimensional coordinates of the key points of the part. Among them, the two-dimensional coordinates can be determined by referring to the above-mentioned method of determining the two-dimensional heat map, which will not be repeated here. In addition, the method of determining the three-dimensional coordinates of the key points of the part and the method of determining the key points of the part are both determined by the method of the heat map. The method of determining the three-dimensional coordinates of the key points of the part is specifically to use the image containing the object in the video frame to be processed as the input of the three-dimensional part key point network model to obtain the three-dimensional heat map of the key points of the part; according to the three-dimensional heat map, the three-dimensional position information of the key points of the part, that is, the three-dimensional coordinates, is determined.

[0173] Since the confidence level represents the probability that an image contains a part key point, the key points that are greater than the confidence level threshold can be called valid part key points, that is, valid part key points are key points within the image. In order to improve the accuracy of 3D posture determination, the valid part key points among the part key points are used, and the invalid key points are not processed.

[0174] When determining the three-dimensional position information of key points of effective parts, a three-dimensional heat map is applied to achieve this, so that the three-dimensional position of the key points obtained is more accurate, and then the three-dimensional posture is determined more accurately.

[0175] In a specific example, for the same part key point, the two elements that characterize its position are the two-dimensional coordinates and the three-dimensional coordinates. In the process of determining the three-dimensional coordinates, the image containing the object in the video frame to be processed is used as the input of the three-dimensional part key point network model to obtain the three-dimensional heat map of the part key point. Assume H1 n For the heat map of the nth part key point with a size of W1×H1×D1, first perform a softmax operation on the heat map to obtain the probability map of the part key point at each position as follows:

[0176]

[0177] After weighted averaging of the probability map, the normalized coordinates of the key points of the part can be obtained. for:

[0178]

[0179]

[0180]

[0181] Among them, H1 is the length of the three-dimensional heat map, W1 is the width of the three-dimensional heat map, and S is the height of the three-dimensional heat map; I is the length number of a pixel point on the three-dimensional heat map, J is the width number of a pixel point on the three-dimensional heat map, and s is the height number of a pixel point on the three-dimensional heat map.

[0182] After determining the confidence of each key point of each part, the three-dimensional posture of the object can be determined. Exemplarily, the three-dimensional posture of the object is determined by solving an optimization problem.

[0183] In S404, when solving the optimization problem, each first posture constraint item is summed to determine the posture optimization item.

[0184] In a specific example, the pose optimization term is the sum of a two-dimensional projection constraint term, a three-dimensional constraint term, and a bone orientation constraint term.

[0185] This involves S405, determining a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

[0186] Specifically, since each constraint item includes multiple parameters, the aforementioned first parameter value can be randomly selected or set according to a certain rule, and the second parameter values ​​of these parameters when the posture optimization item takes the minimum value can be obtained by inversely solving the posture optimization item. Since the second parameter value represents the three-dimensional posture of the object, the second parameter value can be configured to the system or device for determining the three-dimensional posture, thereby realizing the optimization of the three-dimensional posture.

[0187] In a specific example, the parameters to be optimized in the optimization problem are the upper body bone rotation θ, the human body shape coefficient β and the position t of the key point of the set part. In order to distinguish them from the human body model parameters predicted by the network, the human body model parameters predicted by the network model are expressed as θ init , β init ,t init , which is used as the initial value when solving the optimization problem.

[0188] In the optimization process, the posture optimization item is first determined, and the model parameter value of the object's three-dimensional posture when the posture optimization item takes the minimum value is determined as the target rotation angle θ of each bone. fina , human body shape coefficient β fina And set the position of the key point of the part t fina . That is, the second parameter value.

[0189] This method is used to determine the basic posture constraints of the object and the posture optimization items determined. When solving the parameter values ​​of the target model obtained by solving the final problem, the accuracy of the formula is guaranteed, the calculation amount is small, and the calculation speed is faster.

[0190] In the actual application process, in order to improve the accuracy of the optimization process, the second posture constraint item can be introduced on the basis of the posture optimization item composed of the first posture constraint item, and then the posture optimization item is updated. Specifically, each second posture constraint item is determined according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value; each second posture constraint item and the posture optimization item are added together to determine the updated posture optimization item; and the second parameter value of the three-dimensional posture model of the object when the updated posture optimization item takes the minimum value is determined. Since the second posture constraint represents the posture adjustment constraint on the object, the posture adjustment constraint item can correct the unrealizable posture that may exist in the three-dimensional posture of the object determined by the first method, thereby improving the accuracy of the three-dimensional posture determination.

[0191] Exemplarily, the first posture constraint item may include a two-dimensional projection constraint item, a three-dimensional constraint item and a bone orientation constraint item, and the second posture constraint item includes a posture priori constraint item and a posture regularization constraint item. Exemplarily. When determining the three-dimensional posture, the two-dimensional projection constraint item, the three-dimensional constraint item and the bone orientation constraint item can ensure the accuracy of the basic posture determined by the posture when optimizing the problem, and the posture priori constraint item and the posture regularization constraint item can correct the basic posture in the optimization problem, further improving the accuracy of the three-dimensional posture determination.

[0192] The calculation process of each constraint item is explained below:

[0193] (1) Two-dimensional projection constraints:

[0194] If the first posture constraint item is a two-dimensional projection constraint item, the two-dimensional projection constraint item is determined by the following method, including:

[0195] The two-dimensional projection constraint item is determined according to the preset projection constraint item weight, the confidence of each part key point, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each part key point.

[0196] When determining the two-dimensional constraint items, the confidence of the key points of each part is taken into consideration, that is, the selection of the key points of the parts is selective, so that the determined two-dimensional constraint items are more accurate.

[0197] In a specific example, the two-dimensional projection constraint term can be determined by the following formula:

[0198]

[0199] Among them, E 2d represents the two-dimensional projection constraint, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of part key points, K represents the projection matrix, P n (θ, β) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

[0200] Among them, P n In the determination method of (θ, β), according to the characteristics of the SMPL model, θ and β are operated according to the set rules to obtain P n (θ, β). The preset part key point can become the root node, such as the hip. The preset part key point can be used as the part key point of the boundary between the upper body and the lower body in the embodiment of the present application. When the object is the upper body, the key point corresponding to the hip can be used as the preset part key point.

[0201] In the above example, by setting the various parameters in the above formula, such as freely configuring the projection constraint item weight, the accuracy of the determined two-dimensional projection constraint item is improved, thereby making the determined posture optimization item more accurate.

[0202] (2) Three-dimensional constraints:

[0203] If the first posture constraint item is a three-dimensional constraint item, the three-dimensional constraint item is determined by the following method, including:

[0204] The three-dimensional constraint items are determined based on the preset weights of the three-dimensional key point position constraints, the confidence of each key point, the three-dimensional position information of each key point of the part determined according to the model parameters of the object, and the three-dimensional position information of each key point of the part.

[0205] In this way, the confidence of the key points of each part is taken into consideration, that is, the selection of the key points of the part is selective, which makes the determined three-dimensional constraints more accurate.

[0206] In a specific example, the three-dimensional constraint term can be determined by the following formula:

[0207]

[0208] Among them, E 3d represents the three-dimensional projection constraint, ω 3d represents the weight of the pre-set three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of part key points, P n (θ, β)) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the 3D position information of the key point of the nth part. Optional, The three-dimensional position information of the key point of the nth part represented is determined based on the three-dimensional heat map.

[0209] In the above example, by setting the various parameters in the above formula, such as freely configuring the projection constraint item weight, the accuracy of the determined three-dimensional projection constraint item is improved, thereby making the determined posture optimization item more accurate. Effective part key point Effective part key point (3) Skeleton orientation constraint item:

[0210] If the first posture constraint is a bone orientation constraint, the bone orientation constraint is determined in the following manner, including:

[0211] The bone orientation constraint item is determined according to the preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

[0212] In this way, the confidence of each bone and the orientation of the bone are taken into consideration, that is, the selection of bones is selective, which makes the determined bone orientation constraint more accurate.

[0213] In a specific example, the bone orientation constraint can be determined by the following formula:

[0214]

[0215] Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, Dr (θ, β) represents the orientation of each bone determined by the object model parameters, d r The orientation of the rth bone is determined by the key points of the two parts that make up the bone.

[0216] For example, D can be determined by the following formula: j (θ, β):

[0217]

[0218] Among them, P A (θ, β) is the three-dimensional position information of point A determined according to the model parameters of the object, P B (θ, β) is the three-dimensional position information of point B determined according to the model parameters of the object.

[0219] Based on the parameters of each part key point, the confidence of each skeleton can be determined in the following way: the confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the confidence that the skeleton contains valid part key points; the valid part key points are part key points whose confidence is greater than a preset confidence threshold. The confidence of the skeleton is determined by the number of valid part key points constituting the skeleton, for example, the confidence of the skeleton composed of key points that are not drawn is different from the confidence of the skeleton composed of key points that are not drawn, so the subsequent determination of the orientation of the skeleton is more accurate.

[0220] For example, if both of the two part key points constituting the skeleton are valid part key points or one of them is a valid part key point, the confidence of the skeleton can be 1; if both of the two part key points constituting the skeleton are not valid part key points, the confidence of the skeleton can be 0. In this way, the confidence of the corresponding skeleton can be determined according to the confidence of the part key point, that is, the situation that some part key points are out of the picture is taken into account.

[0221] The orientation of the bone can be determined based on the number of valid part key points contained in the two part key points that make up the bone. When determining the orientation of the bone, the number of valid part key points is also taken into account, and the determined orientation of the bone is more accurate.

[0222] Specifically, the bone orientation is the three-dimensional direction of the bones formed by some key point pairs, such as the direction from the elbow to the wrist. In a specific example, the bone orientation can be represented in the form of a three-channel heat map obtained by a pre-trained bone orientation neural network. Assume They are the heat maps of size W×H on the three channels X, Y, and Z of the b-th bone. The two endpoints of the bone are key points A and B, and their two-dimensional coordinates and confidences are c A、c B . First, determine whether the two endpoints are in the picture based on the confidence. If both A and B are not in the picture, the bone orientation is invalid. If only one of A and B is in the picture, the bone orientation is set to the value of the endpoint in the picture. For example, if A is in the picture and B is not in the picture, the bone orientation is:

[0223]

[0224] If both A and B are in the picture, then L points are uniformly sampled on the line connecting A and B, and the bone orientations obtained at these L points are averaged. Specifically, the coordinates of the vth (1≤v≤L)th sampled point can be calculated as The same applies to bones. Finally, for cases where the bone orientation is valid, set the bone orientation Normalized to a 2-norm of 1. Through the above-mentioned judgment method based on the confidence of the key points of the parts, the situation where some joints are outside the screen can be robustly handled.

[0225] In the above example, by configuring the various parameters in the above formula, the determined bone orientation constraint item is more accurate, thereby making the determined posture optimization item more accurate.

[0226] Among them, when the pre-trained skeleton orientation neural network is trained, the skeleton orientation heat map is supervised by the MSE loss function. The supervision of the skeleton orientation heat map can be obtained by the following method: construct a rectangular area connected by endpoints A and B, where the width of the rectangle is fixed to D pixels, and the value of the heat map inside the rectangular area is the supervision value, and the value outside the rectangular area is 0. For example, when endpoints A and B are in the position shown in the figure below, the constructed skeleton orientation heat map supervision can be referred to Figure 5 ,in is the bone orientation heatmap supervision in the X direction, is the bone orientation heatmap supervision in the Y direction, The heatmap supervision of the bone orientation in the Z direction.

[0227] (4) Attitude prior constraints:

[0228] If the second attitude constraint item is an attitude prior constraint item, the attitude prior constraint item is determined by the following method:

[0229] The posture prior constraint item is determined according to the preset posture prior constraint weight, the preset covariance matrix, the preset bone rotation parameter mean and the bone rotation angle parameter in the object model parameter.

[0230] The above-mentioned posture prior constraint items introduce corresponding weights and apply the initial values ​​of the corresponding object model parameters, so as to ensure that the three-dimensional posture solved by the final posture optimization items is not distorted.

[0231] In a specific example, the posture prior constraint term can be determined by the following formula:

[0232] E prior =ω prior ‖A1(θ-b1)‖ 2

[0233] Among them, E prior represents the attitude prior constraint, ω prior represents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

[0234] In the above embodiment, the accuracy of three-dimensional posture determination is improved by freely configuring the parameters in the above formula.

[0235] (5) Posture regularity constraint:

[0236] If the second posture constraint item is a posture regularity constraint item, the posture regularity constraint item is determined by the following method:

[0237] The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

[0238] The above-mentioned attitude regularization constraint item introduces corresponding weights and applies the initial values ​​of the corresponding object model parameters, so as to ensure that the three-dimensional attitude solved by the final attitude optimization item is not distorted.

[0239] In a specific example, the posture regularization constraint can be determined by the following formula:

[0240] E reg =ω pose，reg ‖θ-θ init ‖ 2 +ω shape，reg ‖β-β init ‖ 2

[0241] Among them, E reg represents the attitude regularization constraint, ω pose，reg represents the pre-set posture regression weight, ω shape，regrepresents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, and θ init represents the initial rotation angle value of the object's bones, β represents the object's body shape parameter in the object model parameter, and β init represents the initial coefficient value of the object's body shape. For example, θ init and β init It is determined according to the SMPL model.

[0242] In the above example, the accuracy of three-dimensional posture determination is improved by freely configuring the parameters of the above formula.

[0243] After determining each constraint item, add up each constraint item to obtain the attitude optimization item. Taking the second case above as an example, the attitude optimization item E is:

[0244] E=E 2d +E 3d +E bone +E prior +E reg

[0245] Solve the posture optimization term E and determine the target rotation angle θ of each bone of the object's 3D posture when E takes the minimum value fina , human body shape coefficient β fina And set the position of the key point of the part t fina The determined parameter values ​​are the parameter values ​​corresponding to the three-dimensional posture of the object, so that the three-dimensional posture of the target object can be determined.

[0246] In an embodiment of the present application, in the optimization process of determining the three-dimensional posture, each constraint item is an energy function in the optimization problem, and the weight of each energy function can be configured to improve the effect and robustness of upper body motion capture.

[0247] In addition, it should be noted that the various neural networks used in the aforementioned applications can be independent neural networks or single multi-task neural networks, wherein a single neural network can reduce overhead and increase operating speed.

[0248] like Figure 6 As shown, based on the same inventive concept as the above-mentioned three-dimensional posture determination method, an embodiment of the present application also provides a three-dimensional posture determination device, which includes an image recognition unit 601, a confidence determination unit 602, a basic constraint item determination unit 603, a posture optimization item determination unit 604 and a posture determination unit 605.

[0249] An image recognition unit 601 is configured to recognize an image containing an object in a video frame to be processed;

[0250] A confidence determination unit 602 is configured to determine the confidence of the key points of the parts included in the image and obtain a first parameter value of the three-dimensional posture model of the object;

[0251] A basic constraint item determination unit 603 is configured to process the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents the basic posture constraint of the first parameter value;

[0252] The posture optimization item determination unit 604 is configured to perform summing of the first posture constraint items to determine the posture optimization item;

[0253] The posture determination unit 605 is configured to determine a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

[0254] In some exemplary embodiments, the confidence determination unit 602 is specifically configured to perform:

[0255] Determine a two-dimensional heat map of key points of the parts included in the image;

[0256] For each part key point, the confidence of the part key point is determined according to the resolution of the two-dimensional heat map and the grayscale value of the pixel, where the confidence represents the probability that the image contains the corresponding part key point.

[0257] In some exemplary embodiments, the confidence determination unit 602 is configured to perform:

[0258] Determine the pixel features of the two-dimensional heat map according to the grayscale values ​​of the pixels of the two-dimensional heat map and the resolution of the two-dimensional heat map;

[0259] The pixel features are transformed using a set probability algorithm to obtain the confidence of the key points of the part.

[0260] In some exemplary embodiments, the method further includes an updating unit configured to execute:

[0261] Determine each second posture constraint item according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value;

[0262] Add each second posture constraint item and the posture optimization item to determine an updated posture optimization item;

[0263] A second parameter value of the target model of the three-dimensional pose of the object is determined when the updated pose optimization term takes a minimum value.

[0264] In some exemplary embodiments, the first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item, and a bone orientation constraint item; the second posture constraint item includes at least one of a posture prior constraint item and a posture regularization constraint item.

[0265] In some exemplary embodiments, the posture determination unit 605 is configured to determine the two-dimensional projection constraint item in the following manner:

[0266] The two-dimensional projection constraint item is determined according to the preset projection constraint item weight, the probability parameter of each part key point, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each part key point.

[0267] In some exemplary embodiments, the posture determination unit 605 is configured to determine the two-dimensional projection constraint item by applying the following method:

[0268]

[0269] Among them, E 2d represents the two-dimensional projection constraint, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of part key points, K represents the projection matrix, P n (θ, β)) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

[0270] In some exemplary embodiments, the posture determination unit 605 is configured to determine the three-dimensional constraint item by:

[0271] The three-dimensional constraint items are determined based on the preset weights of the three-dimensional key point position constraints, the confidence of each key point, the three-dimensional position information of each key point of the part determined according to the model parameters of the object, and the three-dimensional position information of each key point of the part.

[0272] In some exemplary embodiments, the posture determination unit 605 is configured to perform the following formula to determine the three-dimensional constraint term:

[0273]

[0274] Among them, E 3d represents the three-dimensional projection constraint, ω 3drepresents the weight of the pre-set three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of part key points, P n (θ, β)) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the three-dimensional position information of the key point of the nth part obtained.

[0275] In some exemplary embodiments, the posture determination unit 605 is configured to determine the skeleton orientation constraint item in the following manner:

[0276] The bone orientation constraint item is determined according to the preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

[0277] In some exemplary embodiments, the posture determination unit 605 is configured to perform the following formula to determine the skeleton orientation constraint:

[0278]

[0279] Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, D r (θ, β) represents the orientation of each bone determined by the object model parameters, d r The orientation of the rth bone is determined by the key points of the two parts that make up the bone.

[0280] In some exemplary embodiments, the posture determination unit 605 is configured to determine the confidence of each bone in the following manner:

[0281] The confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the confidence that the skeleton contains valid part key points; a valid part key point is a part key point whose confidence is greater than a preset confidence threshold.

[0282] In some exemplary embodiments, the posture determination unit 605 is configured to determine the orientation of each bone in the following manner:

[0283] Determines the orientation of the bone based on the number of valid part keypoints contained in the two part keypoints that make up the bone.

[0284] In some exemplary embodiments, the posture determination unit 605 is configured to determine the posture prior constraint item in the following manner:

[0285] The posture prior constraint item is determined according to the preset posture prior constraint weight, the preset covariance matrix, the preset bone rotation parameter mean and the bone rotation angle parameter in the object model parameter.

[0286] In some exemplary embodiments, the posture determination unit 605 is configured to perform the following formula to determine the posture prior constraint:

[0287] E prior =ω prior ‖A1(θ-b1)‖ 2

[0288] Among them, E prior represents the attitude prior constraint, ω prior represents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

[0289] In some exemplary embodiments, the posture determination unit 605 is configured to determine the posture regularization constraint item in the following manner:

[0290] The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

[0291] In some exemplary embodiments, the posture determination unit 605 is configured to perform the following formula to determine the posture regularization constraint:

[0292] E reg =ω pose，reg ‖θ-θ init ‖ 2 +ω shape，reg ‖β-β init ‖ 2

[0293] Among them, E reg represents the attitude regularization constraint, ω pose，reg represents the pre-set posture regression weight, ω shape，reg represents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, and θ init represents the initial rotation angle value of the object's bones, β represents the object's body shape parameter in the object model parameter, and βinit The initial coefficient value representing the object's body shape.

[0294] In some exemplary embodiments, the confidence determination unit 602 is configured to perform:

[0295] The image containing the object in the video to be processed is used as the input of the two-dimensional part key point network model to obtain a two-dimensional heat map of the part key points.

[0296] In some exemplary embodiments, a three-dimensional position information determining unit is further included, and the three-dimensional position information determining unit is configured to determine the three-dimensional position information of the key point of the part by:

[0297] An image containing an object in a video frame to be processed is used as an input of a three-dimensional part key point network model to obtain a three-dimensional heat map of the part key points;

[0298] According to the three-dimensional heat map, the three-dimensional position information of the key points of the part is determined.

[0299] The three-dimensional posture determination device proposed in the embodiment of the present application adopts the same inventive concept as the above-mentioned three-dimensional posture determination method and can achieve the same beneficial effects, which will not be repeated here.

[0300] In some possible implementations, the electronic device according to the present application may include at least one processor and at least one memory. The memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the interface information control method according to various exemplary implementations of the present application described above in this specification. For example, the processor may execute the steps in the interface information control method.

[0301] Refer to the following Figure 7 The electronic device 130 according to this embodiment of the present application is described. Figure 7 The electronic device 130 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0302] like Figure 7 As shown, the electronic device 130 is in the form of a general electronic device. The components of the electronic device 130 may include but are not limited to: the at least one processor 131, the at least one memory 132, and a bus 133 connecting different system components (including the memory 132 and the processor 131).

[0303] Bus 133 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0304] The memory 132 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1321 and / or a cache memory 1322 , and may further include a read-only memory (ROM) 1323 .

[0305] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, such program modules 1324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0306] The electronic device 130 may also communicate with one or more external devices 134 (e.g., keyboards, pointing devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 130, and / or communicate with any device that enables the electronic device 130 to communicate with one or more other electronic devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 135. Furthermore, the electronic device 130 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 136. As shown, the network adapter 136 communicates with other modules for the electronic device 130 via a bus 133. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0307] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 132 including instructions, and the instructions can be executed by the processor 131 of the device 600 to complete the above method. Alternatively, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0308] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by the processor 131, any method of the interface information control method provided in the present application is implemented.

[0309] In an exemplary embodiment, various aspects of an interface information control method provided by the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of an interface information control method according to various exemplary embodiments of the present application described above in this specification.

[0310] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0311] The program product for image zooming of the embodiment of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device. However, the program product of the present application is not limited to this. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0312] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0313] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0314] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the user electronic device, partially on the user device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or entirely on the remote electronic device or server. In the case of a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (for example, using an Internet service provider to connect through the Internet).

[0315] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0316] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0317] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0318] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable image scaling device to produce a machine, so that the instructions executed by the processor of the computer or other programmable image scaling device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0319] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable image scaling device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, the instruction device being implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0320] These computer program instructions may also be loaded onto a computer or other programmable image scaling device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0321] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0322] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining a three-dimensional posture, characterized in that: include: Identifying an image containing an object in a video frame to be processed; Determine the confidence of the key points of the parts included in the image, and obtain the first parameter value of the three-dimensional posture model of the object; wherein the confidence of each key point of the parts is determined by: determining a two-dimensional heat map of the key points of the parts included in the image; for each key point of the parts, determine the confidence of the key point of the parts according to the resolution of the two-dimensional heat map and the gray value of the pixel point, wherein the confidence represents the probability that the image contains the corresponding key point of the parts; Processing the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value; Adding up the first posture constraint items to determine a posture optimization item; Determine a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

2. The method according to claim 1, characterized in that: The step of determining the confidence of the key points of the part according to the resolution of the two-dimensional thermal map and the grayscale value of the pixel point includes: Determining pixel features of the two-dimensional heat map according to grayscale values ​​of pixels of the two-dimensional heat map and a resolution of the two-dimensional heat map; The pixel features are transformed using a set probability algorithm to obtain the confidence of the key points of the part.

3. The method according to claim 1, characterized in that Also includes: Determine each second posture constraint item according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value; Add each of the second posture constraint items and the posture optimization item to determine an updated posture optimization item; Determine a second parameter value of the three-dimensional posture model of the object when the updated posture optimization item takes a minimum value.

4. The method according to claim 3, characterized in that The first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item and a bone orientation constraint item; the second posture constraint item includes at least one of a posture priori constraint item and a posture regularization constraint item.

5. The method according to claim 4, characterized in that The two-dimensional projection constraint is determined as follows: The two-dimensional projection constraint item is determined based on the preset projection constraint item weight, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each of the part key points.

6. The method according to claim 5, characterized in that The two-dimensional projection constraint is determined using the following formula: Among them, E 2d represents the two-dimensional projection constraint term, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of the part key points, K represents the projection matrix, P n (θ, β) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

7. The method according to claim 4, characterized in that The three-dimensional constraint items are determined as follows: The three-dimensional constraint item is determined based on the preset weight of the three-dimensional part key point position constraint item, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the three-dimensional position information of each of the part key points.

8. The method according to claim 7, characterized in that The three-dimensional constraint term is determined using the following formula: Among them, E 3d represents the three-dimensional constraint term, ω 3d represents the weight of the preset three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of the part key points, P n (θ, β) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the three-dimensional position information of the key point of the nth part obtained.

9. The method according to claim 4, characterized in that The bone orientation constraint is determined as follows: The bone orientation constraint item is determined according to a preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

10. The method according to claim 9, characterized in that The following formula is used to determine the bone orientation constraint: Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, D r (θ, β) represents the orientation of each bone determined by the object model parameters, d r is the orientation of the rth bone determined by the two key points of the skeleton.

11. The method according to claim 9, characterized in that The confidence of each of the bones is determined by: The confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the probability that the skeleton contains the valid part key point; the valid part key point is a part key point whose confidence is greater than a preset confidence threshold.

12. The method according to claim 11, characterized in that The orientation of each of the bones is determined as follows: The orientation of the skeleton is determined based on the number of valid part key points included in the two part key points constituting the skeleton.

13. The method according to claim 4, characterized in that The posture prior constraint item is determined by: The posture prior constraint item is determined according to a preset posture prior constraint weight, a preset covariance matrix, a preset bone rotation parameter mean and a bone rotation angle parameter in the object model parameter.

14. The method according to claim 13, characterized in that The following formula is used to determine the posture prior constraint: E prior =ω prior ‖A1(θ-b1)‖ 2 Among them, E prior represents the posture prior constraint, ω prior represents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

15. The method according to claim 4, characterized in that The posture regularization constraint is determined as follows: The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

16. The method according to claim 15, characterized in that The posture regularization constraint is determined by applying the following formula: E reg =ω pose,reg ‖θ-θ init ‖ 2 +oh shape,reg ‖BB init ‖ 2 Among them, E reg represents the attitude regularization constraint, ω pose,reg represents the pre-set posture regression weight, ω shape,reg represents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, θ init represents the initial rotation angle value of the skeleton of the object, β represents the object body parameter in the object model parameter, β init An initial coefficient value representing the object's body shape.

17. The method according to any one of claims 1 to 16, characterized in that: The step of determining a two-dimensional heat map of key points of parts included in the image comprises: An image containing an object in the video to be processed is used as an input of a two-dimensional part key point network model to obtain a two-dimensional heat map of the part key points.

18. The method according to any one of claims 5 to 8, characterized in that: The three-dimensional position information of the key points of the part is determined by: Using the image containing the object in the video frame to be processed as the input of the three-dimensional part key point network model to obtain a three-dimensional heat map of the part key points; According to the three-dimensional heat map, the three-dimensional position information of the key points of the part is determined.

19. A device for determining a three-dimensional posture, characterized in that: include: An image recognition unit is configured to recognize an image containing an object in a video frame to be processed; A confidence determination unit is configured to determine the confidence of the part key points included in the image and obtain a first parameter value of the three-dimensional posture model of the object; wherein the confidence of each part key point is determined by: determining a two-dimensional heat map of the part key points included in the image; for each part key point, determining the confidence of the part key point according to the resolution of the two-dimensional heat map and the gray value of the pixel point, wherein the confidence represents the probability that the image contains the corresponding part key point; A basic constraint item determination unit is configured to process the posture information of the object according to the confidence of the key points of the parts to determine each first posture constraint item, wherein the first posture constraint item represents a basic posture constraint of the first parameter value; a posture optimization item determination unit, configured to perform summing of the first posture constraint items to determine a posture optimization item; The posture determination unit is configured to determine a second parameter value of the three-dimensional posture model of the object when the posture optimization item takes a minimum value; wherein the second parameter value represents the three-dimensional posture of the object.

20. The device according to claim 19, characterized in that The confidence determination unit is configured to perform: Determining pixel features of the two-dimensional heat map according to grayscale values ​​of pixels of the two-dimensional heat map and a resolution of the two-dimensional heat map; The pixel features are transformed using a set probability algorithm to obtain the confidence of the key points of the part.

21. The device according to claim 19, characterized in that Also included is an update unit, the update unit being configured to execute: Determine each second posture constraint item according to the acquired object model parameters, wherein the second posture constraint item represents the posture adjustment constraint of the first parameter value; Add each of the second posture constraint items and the posture optimization item to determine an updated posture optimization item; Determine a second parameter value of the target model of the three-dimensional posture of the object when the updated posture optimization item takes a minimum value.

22. The device according to claim 21, characterized in that The first posture constraint item includes at least one of a two-dimensional projection constraint item, a three-dimensional constraint item and a bone orientation constraint item; the second posture constraint item includes at least one of a posture priori constraint item and a posture regularization constraint item.

23. The device according to claim 22, characterized in that The posture determination unit is configured to determine the two-dimensional projection constraint item in the following manner: The two-dimensional projection constraint item is determined based on the preset projection constraint item weight, the probability parameter of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the two-dimensional position information of each of the part key points.

24. The device according to claim 23, characterized in that The posture determination unit is configured to determine the two-dimensional projection constraint item by applying the following method: Among them, E 2d represents the two-dimensional projection constraint term, ω 2d represents the pre-set projection constraint weight, c n represents the confidence of the nth part key point, N represents the total number of the part key points, K represents the projection matrix, P n (θ, β) is the three-dimensional position information of each key point of the part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; t is the position parameter of the preset key point of the object in the object model parameters; is the two-dimensional position information of the key point of the nth part obtained.

25. The device according to claim 22, characterized in that The posture determination unit is configured to determine the three-dimensional constraint item by: The three-dimensional constraint item is determined based on the preset weight of the three-dimensional part key point position constraint item, the confidence of each of the part key points, the three-dimensional position information of each part key point determined according to the model parameters of the object, and the three-dimensional position information of each of the part key points.

26. The device according to claim 25, characterized in that The posture determination unit is configured to perform the following formula to determine the three-dimensional constraint item: Among them, E 3d represents the three-dimensional constraint term, ω 3d represents the weight of the preset three-dimensional key point position constraint item, c n represents the confidence of the nth part key point, N represents the total number of the part key points, P n (θ, β) is the three-dimensional position information of the key point of the nth part determined according to the model parameters of the object, θ is the bone rotation angle parameter in the object model parameters; β is the object body shape parameter in the object model parameters; is the three-dimensional position information of the key point of the nth part obtained.

27. The device according to claim 22, characterized in that The posture determination unit is configured to determine the skeleton orientation constraint item in the following manner: The bone orientation constraint item is determined according to a preset bone orientation constraint item weight, the confidence of each bone determined according to the confidence, the orientation of each bone and the orientation of each bone determined according to the object model parameters.

28. The device according to claim 27, characterized in that The posture determination unit is configured to apply the following formula to determine the bone orientation constraint item: Among them, E bone Represents the bone orientation constraint, ω bone Indicates the preset bone orientation constraint weight, b r represents the confidence of the rth bone, D r (θ, β) represents the orientation of each bone determined by the object model parameters, d r is the orientation of the rth bone determined by the two key points of the skeleton.

29. The device according to claim 28, characterized in that The posture determination unit is configured to determine the confidence of each of the bones in the following manner: The confidence of the skeleton is determined based on the number of valid part key points contained in the two part key points constituting the skeleton, wherein the confidence of the skeleton represents the confidence that the skeleton contains the valid part key points; the valid part key points are part key points whose confidence is greater than a preset confidence threshold.

30. The device according to claim 29, characterized in that The posture determination unit is configured to determine the orientation of each of the bones in the following manner: The orientation of the skeleton is determined based on the number of valid part key points included in the two part key points constituting the skeleton.

31. The device according to claim 22, characterized in that The posture determination unit is configured to determine the posture priori constraint item by: The posture prior constraint item is determined according to a preset posture prior constraint weight, a preset covariance matrix, a preset bone rotation parameter mean and a bone rotation angle parameter in the object model parameter.

32. The device according to claim 31, characterized in that The posture determination unit is configured to apply the following formula to determine the posture prior constraint item: E prior =ω prior ‖A1(θ-b1)‖ 2 Among them, E prior represents the posture prior constraint, ω prior represents the preset posture prior constraint weight, A1 represents the preset covariance matrix, b1 represents the preset bone rotation parameter mean, and θ is the bone rotation angle parameter in the object model parameter.

33. The device according to claim 32, characterized in that The posture determination unit is configured to determine the posture regularity constraint item in the following manner: The posture regularization constraint item is determined according to the preset posture regression weight, the preset body shape regression weight, the bone rotation angle parameters in the object model parameters, the object body shape parameters in the object model parameters, the initial rotation angle value of the object's bones, and the initial coefficient value of the object's body shape.

34. The device according to claim 33, characterized in that The posture determination unit is configured to perform the following formula to determine the posture regularization constraint item: E reg =ω pose,reg ‖θ-θ init ‖ 2 +oh shape,reg ‖BB init ‖ 2 Among them, E reg represents the attitude regularization constraint, ω pose,reg represents the pre-set posture regression weight, ω shape,reg represents the preset body shape regression weight, θ represents the bone rotation angle parameter in the object model parameter, θ init represents the initial rotation angle value of the skeleton of the object, β represents the object body parameter in the object model parameter, β init An initial coefficient value representing the object's body shape.

35. The device according to any one of claims 19 to 34, characterized in that: The confidence determination unit is configured to perform: An image containing an object in the video to be processed is used as an input of a two-dimensional part key point network model to obtain a two-dimensional heat map of the part key points.

36. The device according to any one of claims 23 to 26, characterized in that The system further includes a three-dimensional position information determining unit, wherein the three-dimensional position information determining unit is configured to determine the three-dimensional position information of the key point of the part by: Using the image containing the object in the video frame to be processed as the input of the three-dimensional part key point network model to obtain a three-dimensional heat map of the part key points; According to the three-dimensional heat map, the three-dimensional position information of the key points of the part is determined.

37. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method for determining a three-dimensional posture as described in any one of claims 1 to 18.

38. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for determining a three-dimensional posture as claimed in any one of claims 1 to 18.

39. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for determining a three-dimensional posture according to any one of claims 1 to 18.

Images