Virtual object driving method, device, storage medium and computer equipment

Abstract

The embodiments of the present application disclose a virtual object driving method, apparatus, storage medium and computer equipment. The embodiments of the present application obtain a target monocular image and extract target image feature information corresponding to the target monocular image; input the target image feature information into a first target preset decoder and a second target preset decoder respectively, and output corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information; input the target two-dimensional heat map information and target three-dimensional skeleton direction field information into a target parameter regressor, and output corresponding target joint rotation parameters; and drive the virtual object based on the target joint rotation parameters. In this way, the corresponding target joint rotation parameters can be quickly output according to the actions of the characters in the target monocular image to drive the virtual object to perform corresponding actions, which greatly improves the efficiency of virtual object driving.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a virtual object driving method, apparatus, storage medium, and computer equipment. Background Art

[0002] With the booming interactive entertainment industry and the development of computer graphics and vision technologies, virtual live streaming technology has become increasingly popular in the live streaming field. Motion capture of human limbs is a crucial part of the entire virtual live streaming process.

[0003] In related technologies, the generation of virtual objects requires that the reconstruction results be consistent with real humans and contain many details, so complete motion capture of the target object is required. However, when generating virtual objects through monocular images, it is very difficult due to the lack of a large amount of input information. Summary of the Invention

[0004] Embodiments of the present application provide a virtual object driving method, apparatus, storage medium, and computer device, which can improve the efficiency of virtual object driving.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] A virtual object driving method, comprising:

[0007] Acquire a target monocular image and extract target image feature information corresponding to the target monocular image;

[0008] Input the target image feature information into the first target preset decoder and the second target preset decoder respectively, and output the corresponding target two-dimensional heat map information and target three-dimensional bone direction field information;

[0009] The first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information, and the labeled two-dimensional heat map information; the second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted from the image feature information and the labeled three-dimensional skeleton direction field information;

[0010] Input the target two-dimensional heat map information and the target three-dimensional bone direction field information into the target parameter regressor, and output the corresponding target joint rotation parameters;

[0011] The target parameter regressor is trained by using the target two-dimensional heat map information and the target three-dimensional bone direction field information, and the predicted joint rotation parameters and label key rotation parameters;

[0012] The virtual object is driven based on the target joint rotation parameter.

[0013] A virtual object driving device, comprising:

[0014] An extraction unit, configured to obtain a target monocular image and extract target image feature information corresponding to the target monocular image;

[0015] A first output unit is configured to input the target image feature information into a first target preset decoder and a second target preset decoder, respectively, and output corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information;

[0016] The first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information, and the labeled two-dimensional heat map information; the second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted from the image feature information and the labeled three-dimensional skeleton direction field information;

[0017] A second output unit is configured to input the target two-dimensional heat map information and the target three-dimensional bone direction field information into a target parameter regressor and output corresponding target joint rotation parameters;

[0018] The target parameter regressor is trained by using the target two-dimensional heat map information and the target three-dimensional bone direction field information, and the predicted joint rotation parameters and label key rotation parameters;

[0019] A driving unit is configured to drive the virtual object based on the target joint rotation parameter.

[0020] In some embodiments, the apparatus further comprises:

[0021] An acquisition unit, configured to acquire a monocular image sample and extract image feature information corresponding to the monocular image sample;

[0022] a third output unit, configured to input the image feature information into a first preset decoder and output corresponding two-dimensional heat map information;

[0023] a first calculation unit, configured to calculate a first difference between the two-dimensional heat map information and the label two-dimensional heat map information;

[0024] a fourth output unit, configured to input the image feature information into a second preset decoder and output corresponding three-dimensional bone direction field information;

[0025] A second calculation unit is used to calculate a second difference between the three-dimensional skeleton direction field information and the label three-dimensional skeleton direction field information;

[0026] A first iterative adjustment unit is configured to iteratively adjust network parameters of the first preset decoder and the second preset decoder according to the first difference and the second difference, and return to execute inputting the image feature information into the first preset decoder and outputting corresponding two-dimensional heat map information until the first difference and the second difference converge to obtain a first target preset decoder and a second target preset decoder.

[0027] In some embodiments, the first iterative adjustment unit is configured to:

[0028] Weighting the first difference by a first weight to obtain a first target loss;

[0029] Weighting the second difference by a second weight to obtain a second target loss;

[0030] Constructing a third target loss based on the first target loss and the second target loss;

[0031] Iteratively adjust the network parameters of the first preset decoder and the second preset decoder according to the third target loss, and return to execute the input of the image feature information to the first preset decoder and output of the corresponding two-dimensional heat map information until the third target loss converges to obtain the first target preset decoder and the second target preset decoder.

[0032] In some embodiments, the apparatus further comprises:

[0033] A fifth output unit, configured to input the target two-dimensional heat map information and the target three-dimensional skeleton direction field information into a parameter regressor, and output corresponding human body shape parameters and joint rotation parameters;

[0034] a first determining unit, configured to determine a three-dimensional joint point according to the human body shape parameters and the joint rotation parameters;

[0035] A projection unit, configured to project the three-dimensional joint points onto a two-dimensional image plane according to a preset function to obtain two-dimensional key points;

[0036] A third calculation unit is used to calculate a third difference between the three-dimensional joint point and the labeled three-dimensional joint point, and a fourth difference between the two-dimensional key point and the labeled two-dimensional key point;

[0037] The second iterative adjustment unit is used to iteratively adjust the network parameters of the parameter regressor based on the third difference and the fourth difference, and return the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, output the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge to obtain the target parameter regressor.

[0038] In some embodiments, the first determining unit is configured to:

[0039] Get a preset parametric human body model;

[0040] The preset parameterized human body model is deformed according to the human body shape parameters and joint rotation parameters to determine the three-dimensional joint points.

[0041] In some embodiments, the second iterative adjustment unit is configured to:

[0042] Calculating a fifth difference between the human body shape parameter and the label shape parameter and a sixth difference between the joint rotation parameter and the label rotation parameter;

[0043] combining the fifth difference and the sixth difference to form a seventh difference;

[0044] generating a priori restricted loss of the joint rotation parameters;

[0045] Weighting the third difference by a third weight to obtain a fourth target loss;

[0046] Weighting the fourth difference by a fourth weight to obtain a fifth target loss;

[0047] weighting the seventh difference by a fifth weight to obtain a sixth target loss;

[0048] Weighting the prior constraint loss by a sixth weight to obtain a seventh target loss;

[0049] constructing an eighth target loss based on the fourth target loss, the fifth target loss, the sixth target loss, and the seventh target loss;

[0050] The network parameters of the parameter regressor are iteratively adjusted according to the eighth target loss, and the target two-dimensional heat map information and the target three-dimensional bone direction field information are returned to the parameter regressor, and the corresponding human body shape parameters and joint rotation parameters are output until the eighth target loss converges to obtain the target parameter regressor.

[0051] In some embodiments, the extraction unit is used to:

[0052] Acquire a monocular image of the target;

[0053] The target monocular image is input into a preset image encoder to extract the corresponding target image feature information.

[0054] In some embodiments, the driving unit is configured to:

[0055] Determining a root joint point of the virtual object;

[0056] Starting from the root joint point, the virtual object is driven according to the target joint rotation parameter and the joint point driving sequence.

[0057] A computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the steps in the above-mentioned virtual object driving method.

[0058] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the steps in the virtual object driving method provided above are implemented.

[0059] A computer program product or computer program includes computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions, causing the computer device to perform the steps of the virtual object driving method provided above.

[0060] The embodiment of the present application obtains a target monocular image and extracts target image feature information corresponding to the target monocular image; inputs the target image feature information into a first target preset decoder and a second target preset decoder, respectively, to output corresponding target two-dimensional heat map information and target three-dimensional skeletal direction field information; inputs the target two-dimensional heat map information and target three-dimensional skeletal direction field information into a target parameter regressor, outputs corresponding target joint rotation parameters; and drives a virtual object based on the target joint rotation parameters. In this way, the corresponding target joint rotation parameters can be quickly output to drive the virtual object to perform corresponding actions based on the actions of the character in the target monocular image, greatly improving the efficiency of virtual object driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 is a schematic diagram of a scenario of a virtual object driving system provided in an embodiment of the present application;

[0063] Figure 2 Schematic diagram of a virtual object driving method according to an embodiment of the present application;

[0064] Figure 3 is another flowchart of the virtual object driving method provided in an embodiment of the present application;

[0065] Figure 4a A two-dimensional human body schematic diagram of the virtual object driving method provided in an embodiment of the present application;

[0066] Figure 4b A schematic diagram of a three-dimensional human body in accordance with a virtual object driving method provided in an embodiment of the present application;

[0067] Figure 4c Schematic diagram of a scenario of a virtual object driving method provided in an embodiment of the present application;

[0068] Figure 5 is a structural diagram of a virtual object driving device provided in an embodiment of the present application;

[0069] Figure 6 It is a structural diagram of the computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0071] Embodiments of the present application provide a virtual object driving method, apparatus, storage medium, and computer equipment.

[0072] See also Figure 1 , Figure 1 A schematic diagram of a scenario for a virtual object driving system provided in an embodiment of the present application includes: a viewer client A and a server B. Viewer client A and server B are connected via a communication network. The communication network may include wireless networks and wired networks, wherein wireless networks include one or more combinations of wireless wide area networks, wireless local area networks, wireless metropolitan area network, and wireless personal area networks. The network includes network entities such as routers and gateways, which are not shown in the figure. Viewer client A can exchange information with server B via the communication network.

[0073] The terminal A can be a terminal with a storage unit and a microprocessor installed and having computing capabilities, such as a tablet computer, a mobile phone, a laptop computer, or a desktop computer. The terminal can be installed with a client A, such as a live broadcast client or a game client. The client A can capture the target monocular image of the current user through a camera and send the target monocular image to the server B.

[0074] The virtual object driving system may include a server B, which may store the correspondence between the host client and each live broadcast room. When a viewer client A selects a live broadcast room, server B transmits the live video stream corresponding to the host client to all viewer clients A within the same live broadcast room based on the correspondence between each live broadcast room and the host client. Server B may be used to store the target monocular image sent by viewer client A.

[0075] The virtual object driving system may include a virtual object driving device, which may be integrated into a server B having a storage unit and a microprocessor installed therein and having computing power. Figure 1 In the embodiment, the server B can be used to obtain a target monocular image and extract target image feature information corresponding to the target monocular image; the target image feature information is input into a first target preset decoder and a second target preset decoder respectively, and the corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information are output; wherein, the first target preset decoder is trained by the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information and the labeled two-dimensional heat map information, and the second target preset decoder is trained by the three-dimensional skeleton direction field information predicted by the image feature information and the labeled three-dimensional skeleton direction field information; the target two-dimensional heat map information and the target three-dimensional skeleton direction field information are input into a target parameter regressor, and the corresponding target joint rotation parameters are output; wherein, the target parameter regressor is trained by the target two-dimensional heat map information and the target three-dimensional skeleton direction field information, the predicted joint rotation parameters and the labeled key rotation parameters; and the virtual object is driven based on the target joint rotation parameters.

[0076] It should be noted that Figure 1 The scene diagram of the virtual object driving system shown is only an example. The virtual object driving system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the virtual object driving system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0077] The following are detailed descriptions of each.

[0078] In this embodiment, the description will be made from the perspective of a virtual object driving device, which can be specifically integrated into a viewer client.

[0079] See also Figure 2 , Figure 2: is a flow chart of a virtual object driving method provided in an embodiment of the present application. The virtual object driving method includes:

[0080] In step 101, a target monocular image is acquired, and target image feature information corresponding to the target monocular image is extracted.

[0081] Among them, the audience client can enter the live broadcast room corresponding to any anchor client, open the live broadcast room corresponding to the anchor client to watch the live broadcast, and the audience client can watch the live broadcast content of the live broadcast user in the live broadcast room, such as talent show, game live broadcast and happy fight, etc., and can also follow the anchor user, give gifts to the anchor user or send interactive messages, etc. to realize interactive operations with the anchor user.

[0082] Accordingly, the anchor client can use virtual live broadcast technology to broadcast live. The virtual live broadcast is to make some actions of the anchor to control the virtual character to do the same action to achieve interesting live broadcast. Based on this, the target monocular image collected by the anchor client in real time can be obtained. The monocular image is a single-frame image. The format of the image can be Graphics Interchange Format (GIF) or bitmap (BMP), etc., which is not specifically limited here.

[0083] Furthermore, the server can also extract target image feature information corresponding to the target monocular image. The extraction method can be through a convolutional neural network model (CNN) or a recurrent neural network model (RNN). The target image feature information can better express the image meaning of the target monocular image to be used subsequently.

[0084] In step 102, the target image feature information is input into the first target preset decoder and the second target preset decoder respectively, and the corresponding target two-dimensional heat map information and target three-dimensional bone direction field information are output.

[0085] Among them, the first target preset decoder and the second target preset decoder can be recurrent neural network models. It should be noted that the first target preset decoder and the second target preset decoder are both trained network models.

[0086] The first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information and the labeled two-dimensional heat map information. The second target preset decoder is obtained by training the three-dimensional bone direction field information predicted by the image feature information and the labeled three-dimensional bone direction field information. The monocular image sample is the image sample used for training. The monocular image sample can carry labeled two-dimensional heat map information and labeled three-dimensional bone direction field information. The two-dimensional heat map information is the two-dimensional position information of the key point in the image coordinate system corresponding to the monocular image sample. The three-dimensional bone direction field information is the direction information of the bone in the spatial coordinate system corresponding to the monocular image sample. The labeled two-dimensional heat map information is the standard two-dimensional heat map information. The labeled three-dimensional bone direction field is the standard three-dimensional bone direction field information. The labeled two-dimensional heat map information and the labeled three-dimensional bone direction field information can be manually preset for calibration to realize backpropagation training of the preset encoder and adjust the network parameters.

[0087] In one embodiment, the mathematical expression of the two-dimensional heat map information can be:

[0088] heatmap μ =exp((x-μ) 2 / 2σ 2 )

[0089] The heatmap μ is the heat value corresponding to the x-coordinate of any point on the heat map corresponding to the key point μ. The exp() is an exponential function with the natural constant e as the base, and the σ can be a constant.

[0090] In one embodiment, the mathematical expression of the three-dimensional skeleton direction field information can be:

[0091] POF xy =(xy) / ‖xy‖

[0092] The POF xy In the figure, x and y are the 3D coordinates of the two end points of the skeleton respectively, and the POF values ​​corresponding to all pixels in the xy skeleton area are POF. xy .

[0093] In some embodiments, the training process may be as follows:

[0094] (1) Obtain a monocular image sample and extract image feature information corresponding to the monocular image sample;

[0095] (2) inputting the image feature information into a first preset decoder, and outputting corresponding two-dimensional heat map information;

[0096] (3) calculating a first difference between the two-dimensional heat map information and the label two-dimensional heat map information;

[0097] (4) inputting the image feature information into a second preset decoder, and outputting corresponding three-dimensional bone direction field information;

[0098] (5) calculating a second difference between the three-dimensional skeleton direction field information and the label three-dimensional skeleton direction field information;

[0099] (6) Iteratively adjusting network parameters of the first preset decoder and the second preset decoder according to the first difference and the second difference, and returning to execute inputting the image feature information into the first preset decoder and outputting corresponding two-dimensional heat map information until the first difference and the second difference converge to obtain the first target preset decoder and the second target preset decoder.

[0100] Among them, a monocular image sample can be obtained, which is a monocular image used for training. The monocular image sample can contain a large number of monocular images, such as 100 monocular images. The monocular image can carry preset labeled two-dimensional heat map information and labeled three-dimensional bone direction field information to extract image feature information of the monocular image sample.

[0101] It should be noted that the first preset decoder and the second preset decoder can be recurrent neural network models, and the first preset decoder and the second preset decoder are both untrained network models. The first preset decoder is used to predict the two-dimensional heat map information of the monocular image, and the second preset encoder is used to predict the three-dimensional bone direction field information of the monocular image.

[0102] Since the network parameters of the first preset decoder and the second preset decoder are both configured initial network parameters, after the image feature information is input into the first preset decoder and the second preset decoder, the predicted two-dimensional heat map information and three-dimensional bone direction field information are both inaccurate. Therefore, the first difference corresponding to the two-dimensional heat map information and the labeled two-dimensional heat map information can be calculated. The first difference is the difference between the two. The larger the first difference, the less accurate the two-dimensional heat map information prediction is, and the smaller the first difference, the more accurate the two-dimensional heat map information prediction is. The second difference corresponding to the three-dimensional bone direction field information and the labeled three-dimensional bone direction field information can also be calculated. The second difference is the difference between the two. Similarly, the larger the second difference, the less accurate the three-dimensional bone direction field information prediction is, and the smaller the second difference, the more accurate the three-dimensional bone direction field information prediction is.

[0103] Furthermore, back propagation can be performed based on the first difference and the second difference, and the network parameters of the first preset decoder and the second preset decoder can be iteratively adjusted by gradient descent to make the network parameters more and more accurate. In order to verify the above adjustment method, after adjusting the network parameters of the first preset decoder and the second preset decoder, the image feature information is returned to the first preset decoder and the second preset decoder after the network parameter adjustment, and the corresponding new two-dimensional heat map information and three-dimensional bone direction field information are output to continue to calculate the new first difference and the second difference. As the number of training times increases, the first difference and the second difference will become smaller and smaller until the first difference and the second difference tend to converge, which means that the network parameter adjustment of the first preset decoder and the second preset decoder is completed, the first preset decoder learns how to predict accurate two-dimensional heat map information based on image feature information, and the second preset decoder learns how to predict accurate three-dimensional bone direction field information based on image feature information. The training is completed, and the first target preset encoder and the second target preset encoder are obtained.

[0104] Based on the first target preset encoder and the second target preset encoder obtained in the above training process, the target image feature information can be input into the first target preset decoder and the second target preset decoder respectively, and the corresponding accurate target two-dimensional heat map information and target three-dimensional bone direction field information can be output.

[0105] In step 103, the target two-dimensional heat map information and the target three-dimensional bone direction field information are input into the target parameter regressor, and the corresponding target joint rotation parameters are output.

[0106] In related technologies, a virtual human body model can be set with 18 key points and 24 joints. Specifically, the 24 joint points are used to control the virtual character. The target joint rotation parameters can be 72-dimensional joint rotation parameters (a total of 24 joints, each joint corresponds to a three-dimensional coordinate), which is used to control the deformation of the entire virtual character.

[0107] Among them, the target parameter regressor can be a convolutional neural network model or a recurrent neural network model. The parameter regressor is used to predict the joint rotation parameters based on the target two-dimensional heat map information and the target three-dimensional bone direction field information. The target parameter regressor is obtained by training the predicted joint rotation parameters and the label key rotation parameters through the target two-dimensional heat map information and the target three-dimensional bone direction field information. The label key rotation parameters are standard key rotation parameters and can be manually preset for calibration to realize back propagation training of the parameter regressor and adjust the network parameters of the parameter regressor.

[0108] In some embodiments, the training process may be as follows:

[0109] (1) Inputting the target two-dimensional heat map information and the target three-dimensional bone direction field information into a parameter regressor, and outputting the corresponding human body shape parameters and joint rotation parameters;

[0110] (2) determining three-dimensional joint points according to the human body shape parameters and joint rotation parameters;

[0111] (3) Projecting the three-dimensional joint points onto the two-dimensional image plane according to a preset function to obtain two-dimensional key points;

[0112] (4) calculating the third difference between the three-dimensional joint point and the three-dimensional joint point of the label and the fourth difference between the two-dimensional key point and the two-dimensional key point of the label;

[0113] (5) Iteratively adjust the network parameters of the parameter regressor based on the third difference and the fourth difference, and return the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, output the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge, and obtain the target parameter regressor.

[0114] Among them, the target two-dimensional heat map information and the target three-dimensional bone direction field information can be input into an untrained parameter regressor to output the corresponding human body shape parameters and joint rotation parameters. The human body shape parameters are used to control the deformation of the entire human body model, the shape parameters control the body shape such as height, weight, etc., and the joint rotation parameters control body movements such as raising hands and feet.

[0115] It should be noted that the parameter regressor can be a recurrent neural network model, and the parameter regressor is an untrained network model. The parameter regressor is used to predict the human body shape parameters and joint rotation parameters corresponding to the target two-dimensional heat map information and the target three-dimensional bone direction field information.

[0116] Since the network parameters of the parameter regressor are the configured initial network parameters, after the target two-dimensional heat map information and the target three-dimensional bone direction field information are output to the parameter regressor, the predicted human body shape parameters and joint rotation parameters are inaccurate. Therefore, in order to verify whether the human body shape parameters and joint rotation parameters are correct, first, it is necessary to determine the three-dimensional joint points based on the human body shape parameters and joint rotation parameters. Specifically, the standard human body model in the standard posture can be deformed in shape and motion according to the human body shape parameters and joint rotation parameters to obtain the changed human body model, and then 24 three-dimensional joint points are found on the human body model.

[0117] Furthermore, the preset function can be a weak perspective projection function, which is used to convert a three-dimensional image into a two-dimensional image. In this way, the three-dimensional joint points can be projected onto the two-dimensional image plane through the preset function, thereby obtaining the two-dimensional key points on the two-dimensional image plane.

[0118] The labeled 3D joint point is a standard 3D joint point, and the 2D key point is a standard 2D key point. Therefore, a third difference can be calculated between the 3D joint point and the labeled 3D joint point. This third difference is the difference between the two. A larger third difference indicates a less accurate 3D joint point prediction, while a smaller third difference indicates a more accurate 3D joint point prediction. Furthermore, a fourth difference can be calculated between the 2D key point and the labeled 2D joint point. This fourth difference is the difference between the two. Similarly, a larger fourth difference indicates a less accurate 2D key point prediction, while a smaller fourth difference indicates a more accurate 2D key point prediction.

[0119] Furthermore, back propagation can be performed based on the third difference and the fourth difference, and the network parameters of the parameter regressor can be iteratively adjusted by gradient descent to make the network parameters more and more accurate. In order to verify the adjustment degree of the above adjustment method, after adjusting the network parameters of the parameter regressor, return to the parameter regressor after inputting the target two-dimensional heat map information and the target three-dimensional bone direction field information into the network adjustment, output the corresponding new human body shape parameters and joint rotation parameters, repeat the above steps, and continue to calculate the new third difference and fourth difference for network adjustment. As the number of training times increases, the third difference and the fourth difference will become smaller and smaller until the third difference and the fourth difference tend to converge, which means that the network parameter adjustment of the parameter regressor is completed, and the parameter regressor learns how to predict accurate human body shape parameters and joint rotation parameters based on the target two-dimensional heat map information and the target three-dimensional bone direction field information. The training is completed, and the target parameter regressor is obtained.

[0120] Based on the target parameter regressor obtained from the above training process, the target two-dimensional heat map information and the target three-dimensional bone direction field information can be input into the target parameter regressor to output the corresponding accurate target joint rotation parameters and target human body shape parameters. Since virtual objects generally have a predetermined body shape, the target human body shape parameters can be ignored and the target joint rotation parameters can be retained.

[0121] In step 104 , the virtual object is driven based on the target joint rotation parameters.

[0122] In the related art, the existing virtual object driving method (i.e., motion capture method) requires expensive professional equipment, which is difficult for ordinary users to accept, or uses ordinary monocular image input, but the acquisition of motion information is through an iterative optimization process, which cannot utilize the annotation information of joint rotation parameters in the existing three-dimensional human body data set, and is not conducive to the deployment and application of new hardware (such as image processors (graphics processing unit, GPU)) in engineering. How to train a human motion capture model that is friendly to the inference engine and can utilize as much annotation information as possible is an urgent problem. Therefore, the embodiment of the present application proposes an end-to-end model training method that directly obtains the rotation parameters of each joint after inputting a monocular image. The target parameter regressor obtained based on the model training method provided by the embodiment of the present application can directly output the target joint rotation parameters.

[0123] Based on this, since the target joint rotation parameter records the motion information of each of the 24 joint points, the forward kinematics algorithm (FK) can be used. The forward kinematics algorithm means that the movement of the virtual character is that each joint point drives the next joint point to move in sequence. That is, this application can be based on the forward kinematics algorithm and control in sequence according to the joint driving order to realize the simulation of driving the virtual object to perform corresponding actions.

[0124] As can be seen from the above, the embodiment of the present application obtains a target monocular image and extracts target image feature information corresponding to the target monocular image; inputs the target image feature information into a first target preset decoder and a second target preset decoder, respectively, and outputs corresponding target two-dimensional heat map information and target three-dimensional skeletal direction field information; inputs the target two-dimensional heat map information and target three-dimensional skeletal direction field information into a target parameter regressor, and outputs corresponding target joint rotation parameters; and drives a virtual object based on the target joint rotation parameters. In this way, the corresponding target joint rotation parameters can be quickly output to drive the virtual object to perform corresponding actions based on the actions of the character in the target monocular image, greatly improving the efficiency of virtual object driving.

[0125] In this embodiment, the description will be made from the perspective of a virtual object driving device. The virtual object driving device can be specifically integrated into a terminal such as a tablet computer, a mobile phone, etc. that has a storage unit and a microprocessor installed and has computing capabilities. The terminal can open a live broadcast client. In this embodiment of the application, the live broadcast client can be a viewer client.

[0126] See also Figure 3 , Figure 3 This is another flowchart of the virtual object driving method provided in an embodiment of the present application. The method may include:

[0127] In step 201, the server obtains a target monocular image, inputs the target monocular image into a preset image encoder, and extracts corresponding target image feature information.

[0128] The preset image encoder can be a recurrent neural network model, and the server can obtain the target monocular image. Please refer to Figure 4c As shown, Figure 4c This is a scene diagram of the virtual object driving method provided in an embodiment of the present application. The image 40 can be a picture of the anchor's live broadcast, in which the anchor can make specific actions. The camera captures the anchor's image to generate a target monocular image 41, and the target monocular image 41 includes the anchor user.

[0129] Furthermore, the target monocular image 41 is input into a preset image encoder, which extracts corresponding target image feature information. The target image feature information can better represent the image meaning of the host user's action in the target monocular image to be used later.

[0130] In step 202, the server obtains a monocular image sample and extracts image feature information corresponding to the monocular image sample, inputs the image feature information into a first preset decoder, outputs the corresponding two-dimensional heat map information, and calculates the first difference between the two-dimensional heat map information and the labeled two-dimensional heat map information.

[0131] Among them, the server can pre-acquire a large number of monocular image samples, which are monocular images used for training. The monocular image samples can include a large number of monocular images, such as 100 monocular images. The monocular images can carry preset labeled two-dimensional heat map information and labeled three-dimensional bone direction field information, and extract image feature information of each monocular image in the monocular image sample.

[0132] Furthermore, the first preset decoder can be a recurrent neural network model, and the first preset decoder is an untrained network model, and the first preset decoder is used to predict the two-dimensional heat map information of the monocular image.

[0133] Since the first preset decoder is the configured initial network parameter, after the image feature information is input into the first preset decoder, the predicted two-dimensional heat map information is inaccurate. Therefore, the first difference corresponding to the two-dimensional heat map information and the labeled two-dimensional heat map information can be calculated. The first difference is the difference between the two. The larger the first difference, the less accurate the two-dimensional heat map information prediction is, and the smaller the first difference, the more accurate the two-dimensional heat map information prediction is. Specifically, the first difference can be calculated using the following formula:

[0134]

[0135] The L heatmap is the first difference, I is the number of key points, q is the pixel index, w is the weight of different key points, h i (q) represents the heat value corresponding to the qth pixel in the heatmap corresponding to the i-th key point (i.e., two-dimensional heat map information), Represents the true value (i.e., the label 2D heat map information).

[0136] In step 203, the server inputs the image feature information into a second preset decoder, outputs the corresponding 3D skeleton direction field information, and calculates a second difference between the 3D skeleton direction field information and the label 3D skeleton direction field information.

[0137] Among them, the second preset decoder can be a recurrent neural network model, and the second preset decoders are all untrained network models, and the second preset encoder is used to predict the three-dimensional skeleton direction field information of the monocular image.

[0138] Since the network parameters of the second preset decoder are the configured initial network parameters, after the image feature information is input into the second preset decoder, the predicted three-dimensional skeleton direction field information is inaccurate. In this way, the second difference corresponding to the three-dimensional skeleton direction field information and the label three-dimensional skeleton direction field information can be calculated. The second difference is the difference between the two. Similarly, the larger the second difference, the less accurate the prediction of the three-dimensional skeleton direction field information, and the smaller the second difference, the more accurate the prediction of the three-dimensional skeleton direction field information. Specifically, the second difference can be calculated by the following formula:

[0139]

[0140] The L POF is the second difference, U is the number of bones, w is the weight of different bones, and P u (q) represents the POF value corresponding to the qth pixel in the POF corresponding to the uth bone (i.e., the three-dimensional bone direction field information). Represents the true value (i.e., the label three-dimensional bone direction field information).

[0141] In step 204, the server weights the first difference by a first weight to obtain a first target loss, weights the second difference by a second weight to obtain a second target loss, and constructs a third target loss based on the first target loss and the second target loss.

[0142] Among them, the third objective loss can be calculated by the following formula:

[0143] L=w h *L heatmap +w p *L POF

[0144] The w h is the first weight, for example, 0.4. p is the second weight, for example, 0.6. That is, according to the above formula, the first difference can be multiplied by the first weight to obtain the first target loss, the second difference can be multiplied by the second weight to obtain the second target loss, and the first target loss plus the second target loss can be added to construct the third target loss.

[0145] In step 205, the server iteratively adjusts the network parameters of the first preset decoder and the second preset decoder according to the third target loss, and returns to execute inputting the image feature information into the first preset decoder and outputting the corresponding two-dimensional heat map information until the third target loss converges to obtain the first target preset decoder and the second target preset decoder.

[0146] Among them, back propagation can be performed according to the third target loss, and the network parameters of the first preset decoder and the second preset decoder can be iteratively adjusted by gradient descent to make the network parameters more and more accurate. In order to verify the above adjustment method, after adjusting the network parameters of the first preset decoder and the second preset decoder, return to re-execute steps 202, 203 and 204 to continue to input the image feature information into the first preset decoder and the second preset decoder after the network parameter adjustment, and output the corresponding new two-dimensional heat map information and three-dimensional bone direction field information to continue to calculate the new third target loss. As the number of training times increases, the third target loss will become smaller and smaller until the third target loss tends to converge, which means that the network parameter adjustment of the first preset decoder and the second preset decoder is completed, the first preset decoder learns how to predict accurate two-dimensional heat map information based on image feature information, and the second preset decoder learns how to predict accurate three-dimensional bone direction field information based on image feature information. The training is completed, and the first target preset encoder and the second target preset encoder are obtained.

[0147] In step 206 , the server inputs the target image feature information into the first target preset decoder and the second target preset decoder, respectively, and outputs corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information.

[0148] Among them, based on the first target preset encoder and the second target preset encoder obtained in the above training process, the server can input the target image feature information into the first target preset decoder and the second target preset decoder respectively, and output the corresponding accurate target two-dimensional heat map information and target three-dimensional bone direction field information.

[0149] In step 207, the server inputs the target two-dimensional heat map information and the target three-dimensional skeleton direction field information into a parameter regressor, and outputs corresponding human body shape parameters and joint rotation parameters.

[0150] Among them, the target two-dimensional heat map information and the target three-dimensional bone direction field information can be input into an untrained parameter regressor to output corresponding human body shape parameters and joint rotation parameters. The human body shape parameters are used to control the deformation of the entire human body model. The shape parameters control body shapes such as height, weight, and thinness. The joint rotation parameters control body movements such as raising hands and feet. The human body shape parameters can be 10-dimensional human body shape parameters, and the joint rotation parameters can be 72-dimensional joint rotation parameters (24 joints, each corresponding to a three-dimensional coordinate).

[0151] It should be noted that the parameter regressor can be a recurrent neural network model, and the parameter regressor is an untrained network model. The parameter regressor is used to predict the human body shape parameters and joint rotation parameters corresponding to the target two-dimensional heat map information and the target three-dimensional bone direction field information.

[0152] Since the network parameters of the parameter regressor are the configured initial network parameters, after the target two-dimensional heat map information and the target three-dimensional bone direction field information are input into the parameter regressor, the corresponding human body shape parameters and joint rotation parameters are inaccurate.

[0153] In step 208 , the server obtains a preset parameterized human body model, deforms the preset parameterized human body model according to human body shape parameters and joint rotation parameters, and determines three-dimensional joint points.

[0154] Please also refer to Figure 4a and Figure 4b As shown, Figure 4a A two-dimensional human body diagram of the virtual object driving method provided in an embodiment of the present application, Figure 4b A schematic diagram of a 3D human figure in accordance with the virtual object driving method provided in an embodiment of the present application. The virtual 2D human figure 20 can be configured with 18 2D key points. The virtual 3D human figure 30 can be configured with 24 3D joints, each of which is used to control a different part of the 3D virtual human figure 30.

[0155] Since the predicted human body shape parameters and joint rotation parameters are inaccurate, in order to verify whether the human body shape parameters and joint rotation parameters are correct, the server can first obtain a preset parameterized human body model. The preset parameterized human body model T is a standard human body model in a standard posture, such as human body model 30. The standard human body model in the standard posture can be deformed in shape and motion according to the human body shape parameters and joint rotation parameters to obtain a modified human body model, and then 24 three-dimensional joint points are found on the human body model. Specifically, the deformation of the standard human body model in the standard posture can be achieved by the following formula:

[0156] T(β,θ)=T+B S (β)+B P (θ) (1)

[0157] M(β,θ)=W(T(β,θ),J(θ),θ,w) (2)

[0158] T is the standard human body model in standard posture, B S (β) and B P (θ) represents the body shape differences caused by individual differences and human motion, and T(β,θ) represents the 3D human body in the standard posture after body shape deformation. J(θ) represents the 3D coordinates of each joint after the rotation transformation of each joint in the standard posture according to the joint rotation parameters. w is the skinning matrix, W is the linear skinning function, and M(β,θ) represents the 3D human body after body shape deformation and motion deformation. Based on the above formulas (1) and (2), the three-dimensional joint points corresponding to the 24 joint points can be obtained.

[0159] In step 209, the server projects the three-dimensional joint points to the two-dimensional image plane according to a preset function to obtain two-dimensional key points, calculates the third difference between the three-dimensional joint points and the labeled three-dimensional joint points, and calculates the fourth difference between the two-dimensional key points and the labeled two-dimensional key points.

[0160] The preset function can be a weak perspective projection function, which is used to convert a three-dimensional image into a two-dimensional image. In this way, the three-dimensional joint points can be projected onto the two-dimensional image plane through the preset function, thereby obtaining the two-dimensional key points on the two-dimensional image plane. Specifically, the calculation of the two-dimensional key points can be achieved by the following formula:

[0161] K=Π(R·J+t)

[0162] The K is the two-dimensional key point, the Π() is the weak perspective projection function, the R and t are the global rotation and translation, which are known in advance and can also be predicted by the parameter regressor, and the J is the three-dimensional joint point. In this way, the two-dimensional key point can be obtained through the above formula.

[0163] Furthermore, the third difference between the three-dimensional joint point and the labeled three-dimensional joint point can be calculated. The third difference is the difference between the two. The larger the third difference, the less accurate the three-dimensional joint point prediction is, and the smaller the third difference, the more accurate the three-dimensional joint point prediction is. The fourth difference between the two-dimensional key point and the labeled two-dimensional joint point can also be calculated. The fourth difference is the difference between the two. Similarly, the larger the fourth difference, the less accurate the two-dimensional key point prediction is, and the smaller the fourth difference, the more accurate the two-dimensional key point prediction is. Specifically, the third difference can be calculated using the following formula:

[0164]

[0165] The L 2d is the third difference, the K(i) is the two-dimensional key point, I=18, the K * (i) Label the two-dimensional joint points.

[0166] Specifically, the third difference can be calculated by the following formula:

[0167]

[0168] The L 3d is the fourth difference, J(x)-J(y) is the three-dimensional bone direction calculated based on the three-dimensional joint points, and the J * (x)-J * (y) is the true value of the 3D bone direction calculated based on the labeled 3D joint points, where U=24, F cos is the cosine similarity operator.

[0169] In step 210, the server calculates the fifth difference corresponding to the human body shape parameters and the label shape parameters and the sixth difference corresponding to the joint rotation parameters and the label rotation parameters, combines the fifth difference and the sixth difference to form a seventh difference, generates a priori restricted loss for the joint rotation parameters, weights the third difference by the third weight to obtain a fourth target loss, weights the fourth difference by the fourth weight to obtain a fifth target loss, weights the seventh difference by the fifth weight to obtain a sixth target loss, weights the priori restricted loss by the sixth weight to obtain a seventh target loss, and constructs an eighth target loss based on the four target losses, the fifth target loss, the sixth target loss, and the seventh target loss.

[0170] In order to achieve better training results, a fifth difference between the human body shape parameters and the preset standard label human body shape parameters, and a sixth difference between the joint rotation parameters and the preset standard label rotation parameters can be introduced. The fifth difference and the sixth difference are combined to form a seventh difference. Specifically, the seventh difference can be calculated using the following formula:

[0171]

[0172] The L parm This is the seventh difference, where θ(i) is the joint rotation parameter, and θ * (i) is the label rotation parameter. The β(i) is the human body shape parameter, the β * (i) is the label human body shape parameter.

[0173] Furthermore, in order to avoid reverse bending and large-angle bending of the joint, some prior knowledge can be used to generate the prior restriction loss of the joint rotation parameters. The prior restriction loss can be expressed by the following formula:

[0174] L prior =prior(θ)

[0175] The L prior That is the prior restriction loss, and θ is the prior knowledge restriction.

[0176] Among them, the eighth objective loss can be calculated by the following formula:

[0177] l=w 2d *L 2d +w 3d *L 3d +w parm *L parm +w prior *L prior

[0178] The w 2d , w 3d , w parm and w prior are the third weight, fourth weight, fifth weight and sixth weight respectively, and L is the eighth target loss. In this way, the eighth target loss can be calculated by combining the above formula.

[0179] In step 211, the server iteratively adjusts the network parameters of the parameter regressor according to the eighth target loss, and returns the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, outputs the corresponding human body shape parameters and joint rotation parameters, until the eighth target loss converges, and obtains the target parameter regressor.

[0180] Among them, back propagation can be performed according to the eighth target loss, and the network parameters of the parameter regressor can be iteratively adjusted by gradient descent to make the network parameters more and more accurate. In order to verify the adjustment degree of the above adjustment method, after adjusting the network parameters of the parameter regressor, return to execute steps 207, 208, 209 and 210, input the target two-dimensional heat map information and the target three-dimensional bone direction field information into the parameter regressor after network adjustment, output the corresponding new human body shape parameters and joint rotation parameters, and continue to calculate the new eighth target loss for network adjustment. As the number of training times increases, the eighth target loss will become smaller and smaller until the eighth target loss tends to converge, which means that the network parameter adjustment of the parameter regressor is completed, and the parameter regressor learns how to predict accurate human body shape parameters and joint rotation parameters based on the target two-dimensional heat map information and the target three-dimensional bone direction field information. The training is completed, and the target parameter regressor is obtained.

[0181] In step 212, the server inputs the target two-dimensional heat map information and the target three-dimensional bone direction field information into the target parameter regressor, outputs the corresponding target joint rotation parameters, determines the root joint point of the virtual object, and starts from the root joint point to drive the virtual object according to the target joint rotation parameters and the joint point driving sequence.

[0182] Among them, the target parameter regressor obtained based on the above training process can input the target two-dimensional heat map information and the target three-dimensional bone direction field information into the target parameter regressor, and output the corresponding accurate target joint rotation parameters and target human body shape parameters. Since virtual objects generally have a fixed body shape, the target human body shape parameters can be ignored and the target joint rotation parameters can be retained.

[0183] For further information, please refer to Figure 4b Since the target joint rotation parameter records the motion information of each of the 24 joints, the forward kinematics algorithm (FK) can be used. The forward kinematics algorithm means that the movement of the virtual character is that each joint drives the next joint to move in sequence. The joint structure of the human body can be understood as a tree model structure. The rotation order is from the root joint 0 of the virtual object, and the virtual object is driven according to the joint drive order. The joint drive order can be shown in the following Table 1:

[0184] Serial number Parent Node Serial number Parent Node Serial number Parent Node Serial number Parent Node 0 — 6 3 12 9 18 16 1 0 7 4 13 9 19 17 2 0 8 5 14 9 20 18 3 0 9 6 15 12 21 19 4 1 10 7 16 13 22 20 5 2 11 8 17 14 23 21

[0185] Table 1

[0186] Based on Table 1, Table 1 expresses the driving order of the three-dimensional joint points 1, 2 and 3 whose parent nodes are all 0 and 4 whose parent node is 1, which means that the 0 joint point first performs the action to drive the 1, 2, 3 joint points, and then drive the 4 joint point, and so on, which is the joint driving order. That is, this application can be based on the forward motion algorithm, starting from the root joint point, and driving the virtual object according to the target joint rotation parameters and the joint driving order. Please continue to refer to Figure 4c As shown, the virtual object 42 will be driven accordingly according to the host's actions, which greatly enhances the fun of the live broadcast.

[0187] As can be seen from the above, the embodiment of the present application obtains a target monocular image and extracts target image feature information corresponding to the target monocular image; inputs the target image feature information into a first target preset decoder and a second target preset decoder, respectively, and outputs corresponding target two-dimensional heat map information and target three-dimensional skeletal direction field information; inputs the target two-dimensional heat map information and target three-dimensional skeletal direction field information into a target parameter regressor, and outputs corresponding target joint rotation parameters; and drives a virtual object based on the target joint rotation parameters. In this way, the corresponding target joint rotation parameters can be quickly output to drive the virtual object to perform corresponding actions based on the actions of the character in the target monocular image, greatly improving the efficiency of virtual object driving.

[0188] In this embodiment, the description will be made from the perspective of a virtual object driving device, which can be specifically integrated into the anchor client.

[0189] See also Figure 5 , Figure 5 This is a structural diagram of a virtual object driving device provided in an embodiment of the present application. The virtual object driving device is applied to a server, wherein the virtual object driving device may include an extraction unit 301, a first output unit 302, a second output unit 303, and a driving unit 304, etc.

[0190] The extraction unit 301 is configured to obtain a target monocular image and extract target image feature information corresponding to the target monocular image.

[0191] In some embodiments, the extraction unit 301 is configured to:

[0192] Acquire a monocular image of the target;

[0193] The target monocular image is input into a preset image encoder to extract the corresponding target image feature information.

[0194] The first output unit 302 is used to input the target image feature information into the first target preset decoder and the second target preset decoder respectively, and output corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information.

[0195] Among them, the first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information and the labeled two-dimensional heat map information, and the second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted by the image feature information and the labeled three-dimensional skeleton direction field information.

[0196] The second output unit 303 is used to input the target two-dimensional heat map information and the target three-dimensional skeleton direction field information into the target parameter regressor, and output the corresponding target joint rotation parameters.

[0197] Among them, the target parameter regressor is trained by predicting joint rotation parameters and label key rotation parameters through the target two-dimensional heat map information and target three-dimensional bone direction field information.

[0198] The driving unit 304 is configured to drive the virtual object based on the target joint rotation parameter.

[0199] In some embodiments, the driving unit 304 is configured to:

[0200] Determine the root joint point of the virtual object;

[0201] Starting from the root joint point, the virtual object is driven according to the target joint rotation parameters and the joint point driving order.

[0202] In some embodiments, the apparatus further comprises:

[0203] An acquisition unit, configured to acquire a monocular image sample and extract image feature information corresponding to the monocular image sample;

[0204] a third output unit, configured to input the image feature information into a first preset decoder and output corresponding two-dimensional heat map information;

[0205] A first calculation unit, configured to calculate a first difference between the two-dimensional heat map information and the label two-dimensional heat map information;

[0206] A fourth output unit, configured to input the image feature information into a second preset decoder and output corresponding three-dimensional bone direction field information;

[0207] A second calculation unit is used to calculate a second difference between the three-dimensional skeleton direction field information and the label three-dimensional skeleton direction field information;

[0208] The first iterative adjustment unit is used to iteratively adjust network parameters of the first preset decoder and the second preset decoder according to the first difference and the second difference, and return to execute inputting the image feature information into the first preset decoder and outputting corresponding two-dimensional heat map information until the first difference and the second difference converge to obtain the first target preset decoder and the second target preset decoder.

[0209] In some embodiments, the first iterative adjustment unit is configured to:

[0210] The first difference is weighted by a first weight to obtain a first target loss;

[0211] The second difference is weighted by a second weight to obtain a second target loss;

[0212] constructing a third target loss based on the first target loss and the second target loss;

[0213] The network parameters of the first preset decoder and the second preset decoder are iteratively adjusted according to the third target loss, and the image feature information is input into the first preset decoder and the corresponding two-dimensional heat map information is output until the third target loss converges to obtain the first target preset decoder and the second target preset decoder.

[0214] In some embodiments, the apparatus further comprises:

[0215] A fifth output unit is configured to input the target two-dimensional heat map information and the target three-dimensional bone direction field information into a parameter regressor, and output corresponding human body shape parameters and joint rotation parameters;

[0216] A first determining unit is used to determine a three-dimensional joint point according to the human body shape parameter and the joint rotation parameter;

[0217] A projection unit, configured to project the three-dimensional joint points onto a two-dimensional image plane according to a preset function to obtain two-dimensional key points;

[0218] A third calculation unit is used to calculate a third difference between the three-dimensional joint point and the labeled three-dimensional joint point, and a fourth difference between the two-dimensional key point and the labeled two-dimensional key point;

[0219] The second iterative adjustment unit is used to iteratively adjust the network parameters of the parameter regressor based on the third difference and the fourth difference, and return the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, output the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge to obtain the target parameter regressor.

[0220] In some embodiments, the first determining unit is configured to:

[0221] Get a preset parametric human body model;

[0222] The preset parameterized human body model is deformed according to the human body shape parameters and joint rotation parameters to determine the three-dimensional joint points.

[0223] In some embodiments, the second iterative adjustment unit is configured to:

[0224] Calculating a fifth difference between the human body shape parameter and the label shape parameter and a sixth difference between the joint rotation parameter and the label rotation parameter;

[0225] combining the fifth difference and the sixth difference to form a seventh difference;

[0226] Generate a priori constraint loss for the joint rotation parameters;

[0227] The third difference is weighted by a third weight to obtain a fourth target loss;

[0228] The fourth difference is weighted by a fourth weight to obtain a fifth target loss;

[0229] The seventh difference is weighted by the fifth weight to obtain a sixth target loss;

[0230] The prior limit loss is weighted by the sixth weight to obtain the seventh target loss;

[0231] constructing an eighth target loss based on the fourth target loss, the fifth target loss, the sixth target loss, and the seventh target loss;

[0232] The network parameters of the parameter regressor are iteratively adjusted according to the eighth target loss, and the target two-dimensional heat map information and the target three-dimensional bone direction field information are returned to the parameter regressor, and the corresponding human body shape parameters and joint rotation parameters are output until the eighth target loss converges to obtain the target parameter regressor.

[0233] The embodiment of the present application also provides a computer device, which can be a terminal or a server. Figure 6 , which shows a schematic diagram of the structure of a computer device involved in an embodiment of the present invention, specifically:

[0234] The computer device may include one or more processing core processors 601, one or more computer readable storage media memories 602, a power supply 603, an input unit 604 and other components. Those skilled in the art will understand that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0235] Processor 601 is the control center of the computer device. It connects the various components of the entire computer device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 602 and accessing data stored in memory 602, it performs various functions of the computer device and processes data, thereby providing overall monitoring of the computer device. Optionally, processor 601 may include one or more processing cores; preferably, processor 601 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 601.

[0236] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0237] The computer device also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 603 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0238] The computer device may further include an input unit 604, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0239] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the computer device will load the executable files corresponding to one or more application processes into the memory 602 according to the following instructions, and the processor 601 will run the application stored in the memory 602 to implement various functions as follows:

[0240] Obtain a target monocular image and extract target image feature information corresponding to the target monocular image;

[0241] Input the target image feature information into the first target preset decoder and the second target preset decoder respectively, and output the corresponding target two-dimensional heat map information and target three-dimensional bone direction field information;

[0242] The first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information, and the labeled two-dimensional heat map information. The second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted by the image feature information and the labeled three-dimensional skeleton direction field information.

[0243] The target 2D heat map information and the target 3D bone direction field information are input into the target parameter regressor, and the corresponding target joint rotation parameters are output;

[0244] The target parameter regressor is trained by using the target 2D heat map information and target 3D bone orientation field information, and the predicted joint rotation parameters and label key rotation parameters;

[0245] Drive the virtual object based on the target joint rotation parameters.

[0246] In the above embodiments, the description of each embodiment has its own focus. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the virtual object driving method above, which will not be repeated here.

[0247] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0248] To this end, an embodiment of the present application provides a computer storage medium, which stores multiple instructions. The instructions can be loaded by a processor to execute the steps in any virtual object driving method provided in the embodiment of the present application.

[0249] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations provided in the above embodiments.

[0250] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0251] The computer storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0252] Since the instructions stored in the computer storage medium can execute the steps in any virtual object driving method provided in the embodiments of the present application, the beneficial effects that can be achieved by any virtual object driving method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0253] The above is a detailed introduction to a virtual object driving method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A virtual object driving method, characterized in that: include: Acquire a target monocular image and extract target image feature information corresponding to the target monocular image; The target image feature information is input into a first target preset decoder and a second target preset decoder respectively, and the corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information are output; wherein the first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information and the labeled two-dimensional heat map information, and the second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted by the image feature information and the labeled three-dimensional skeleton direction field information; The target two-dimensional heat map information and the target three-dimensional bone direction field information are input into the target parameter regressor, and the corresponding target joint rotation parameters are output; wherein, the target parameter regressor is trained by the joint rotation parameters and the label key rotation parameters predicted by the target two-dimensional heat map information and the target three-dimensional bone direction field information; the training process of the target parameter regressor includes: inputting the target two-dimensional heat map information and the target three-dimensional bone direction field information into the parameter regressor, and outputting the corresponding human body shape parameters and joint rotation parameters; determining the three-dimensional joint points according to the human body shape parameters and the joint rotation parameters; projecting the three-dimensional joint points onto the two-dimensional image plane according to a preset function to obtain the two-dimensional key points; calculating the third difference corresponding to the three-dimensional joint points and the label three-dimensional joint points, and the fourth difference corresponding to the two-dimensional key points and the label two-dimensional key points; iteratively adjusting the network parameters of the parameter regressor based on the third difference and the fourth difference, and returning to inputting the target two-dimensional heat map information and the target three-dimensional bone direction field information into the parameter regressor, outputting the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge to obtain the target parameter regressor; The virtual object is driven based on the target joint rotation parameter.

2. The virtual object driving method according to claim 1, wherein: The method further comprises: Obtaining a monocular image sample and extracting image feature information corresponding to the monocular image sample; Inputting the image feature information into a first preset decoder and outputting corresponding two-dimensional heat map information; Calculating a first difference between the two-dimensional heat map information and the label two-dimensional heat map information; Inputting the image feature information into a second preset decoder and outputting corresponding three-dimensional bone direction field information; Calculating a second difference between the three-dimensional skeleton direction field information and the label three-dimensional skeleton direction field information; Iteratively adjust the network parameters of the first preset decoder and the second preset decoder according to the first difference and the second difference, and return to execute inputting the image feature information into the first preset decoder and outputting the corresponding two-dimensional heat map information until the first difference and the second difference converge to obtain the first target preset decoder and the second target preset decoder.

3. The virtual object driving method according to claim 2, wherein: The iteratively adjusting network parameters of the first preset decoder and the second preset decoder according to the first difference and the second difference, and returning to input the image feature information into the first preset decoder and outputting corresponding two-dimensional heat map information until the first difference and the second difference converge to obtain a first target preset decoder and a second target preset decoder, including: Weighting the first difference by a first weight to obtain a first target loss; Weighting the second difference by a second weight to obtain a second target loss; Constructing a third target loss based on the first target loss and the second target loss; Iteratively adjust the network parameters of the first preset decoder and the second preset decoder according to the third target loss, and return to execute the input of the image feature information to the first preset decoder and output of the corresponding two-dimensional heat map information until the third target loss converges to obtain the first target preset decoder and the second target preset decoder.

4. The virtual object driving method according to claim 1, wherein: The determining of three-dimensional joint points according to the human body shape parameters and joint rotation parameters includes: Get a preset parametric human body model; The preset parameterized human body model is deformed according to the human body shape parameters and joint rotation parameters to determine the three-dimensional joint points.

5. The virtual object driving method according to claim 1, wherein: The method iteratively adjusts the network parameters of the parameter regressor based on the third difference and the fourth difference, and returns the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, outputs the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge, to obtain a target parameter regressor, including: Calculating a fifth difference between the human body shape parameter and the label shape parameter and a sixth difference between the joint rotation parameter and the label rotation parameter; combining the fifth difference and the sixth difference to form a seventh difference; generating a priori constraint loss for the joint rotation parameters; Weighting the third difference by a third weight to obtain a fourth target loss; Weighting the fourth difference by a fourth weight to obtain a fifth target loss; weighting the seventh difference by a fifth weight to obtain a sixth target loss; Weighting the prior constraint loss by a sixth weight to obtain a seventh target loss; constructing an eighth target loss based on the fourth target loss, the fifth target loss, the sixth target loss, and the seventh target loss; The network parameters of the parameter regressor are iteratively adjusted according to the eighth target loss, and the target two-dimensional heat map information and the target three-dimensional bone direction field information are returned to the parameter regressor, and the corresponding human body shape parameters and joint rotation parameters are output until the eighth target loss converges to obtain the target parameter regressor.

6. The virtual object driving method according to claim 1, wherein: The acquiring of a target monocular image and extracting target image feature information corresponding to the target monocular image includes: Acquire a monocular image of the target; The target monocular image is input into a preset image encoder to extract the corresponding target image feature information.

7. The virtual object driving method according to claim 1, wherein: The driving of the virtual object based on the target joint rotation parameter comprises: Determining a root joint point of the virtual object; Starting from the root joint point, the virtual object is driven according to the target joint rotation parameter and the joint point driving sequence.

8. A virtual object driving device, characterized in that: include: An extraction unit, configured to obtain a target monocular image and extract target image feature information corresponding to the target monocular image; A first output unit is configured to input the target image feature information into a first target preset decoder and a second target preset decoder, respectively, and output corresponding target two-dimensional heat map information and target three-dimensional skeleton direction field information; The first target preset decoder is obtained by training the image feature information extracted from the monocular image sample, the predicted two-dimensional heat map information, and the labeled two-dimensional heat map information; the second target preset decoder is obtained by training the three-dimensional skeleton direction field information predicted from the image feature information and the labeled three-dimensional skeleton direction field information; A second output unit is configured to input the target two-dimensional heat map information and the target three-dimensional bone direction field information into a target parameter regressor and output corresponding target joint rotation parameters; The target parameter regressor is trained by using the target two-dimensional heat map information and the target three-dimensional bone direction field information, and the predicted joint rotation parameters and label key rotation parameters; a driving unit, configured to drive the virtual object based on the target joint rotation parameter; A fifth output unit, configured to input the target two-dimensional heat map information and the target three-dimensional skeleton direction field information into a parameter regressor, and output corresponding human body shape parameters and joint rotation parameters; a first determining unit, configured to determine a three-dimensional joint point according to the human body shape parameters and the joint rotation parameters; A projection unit, configured to project the three-dimensional joint points onto a two-dimensional image plane according to a preset function to obtain two-dimensional key points; A third calculation unit is used to calculate a third difference between the three-dimensional joint point and the labeled three-dimensional joint point, and a fourth difference between the two-dimensional key point and the labeled two-dimensional key point; The second iterative adjustment unit is used to iteratively adjust the network parameters of the parameter regressor based on the third difference and the fourth difference, and return the target two-dimensional heat map information and the target three-dimensional bone direction field information to the parameter regressor, output the corresponding human body shape parameters and joint rotation parameters, until the third difference and the fourth difference converge to obtain the target parameter regressor.

9. A computer-readable storage medium, characterized in that The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the virtual object driving method according to any one of claims 1 to 7.

10. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps of the virtual object driving method according to any one of claims 1 to 7 when executing the computer program.

Images