Model penetration correction method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides a model penetration correction method, device, electronic device and storage medium, the method comprising: obtaining the original rotation information matrix and skeleton information matrix of the original human body model; inputting the original rotation information matrix and skeleton information matrix into the redirected model to obtain the redirected rotation information matrix; determining the three-dimensional mesh information of the redirected human body model; calculating the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and determining the similarity between the redirected rotation information matrix and the original rotation information matrix; judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix; if the redirected model has converged, determining that the redirected rotation information matrix represents: the rotation information of the redirected human body model corresponding to the original human body model. The above method reduces labor consumption.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a model penetration correction method, device, electronic device and storage medium. Background Art

[0002] At present, due to the increasingly widespread application of 3D virtual characters in animation production and movie special effects, the production of 3D virtual characters has become increasingly important. At present, the production of 3D virtual characters mainly includes: capturing various actions of the target character through motion capture technology, and then migrating the captured actions to a specified human body model, driving the human body model to perform various actions. In animation production, it is also often necessary to redirect the human body model. Among them, redirecting the human body model refers to: migrating the captured actions to a specified human body model, so that the specified human body model performs corresponding actions. However, there are many problems in the process of migrating the captured actions to the specified human body model. For example, due to the different body shapes of the target character and the human body model to be driven, it may cause the human body model to perform some actions that are not in line with the principles of physics, such as Figure 1 As shown, the driven human model 110 may extend the hand 101 into the limb 102. This situation where the driven human model performs some actions that do not conform to the principles of physics is called "piercing the model". That is, after the target human model is redirected, the redirected target human model may experience the phenomenon of penetrating the model.

[0003] Currently, the penetration of the target human body model after redirection is mainly repaired frame by frame manually. However, the manual frame-by-frame penetration method consumes a lot of manpower and takes a long time, and the repair efficiency is low. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a model penetration correction method, device, electronic device and storage medium to reduce the labor consumption of model penetration repair of the redirected human body model and improve the efficiency of model penetration repair.

[0005] In order to achieve the above object, an embodiment of the present invention provides a model penetration correction method, comprising:

[0006] Acquire an original rotation information matrix and a skeleton information matrix of the original human body model; the original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in T-Pose (T-letter posture);

[0007] Inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix;

[0008] Based on the redirection rotation information matrix, determine the three-dimensional mesh information of the human body model after redirection;

[0009] Calculating the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and determining the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0010] Based on the penetration probability of the human body model after redirection and the similarity between the redirected rotation information matrix and the original rotation information matrix, determine whether the redirected model converges;

[0011] If the redirection model has not converged, the parameters of the redirection model are adjusted, and the process returns to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix;

[0012] If the redirected model has converged, the redirected rotation information matrix is ​​determined to represent: the rotation information of the redirected human body model corresponding to the original human body model.

[0013] Furthermore, judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix includes:

[0014] If the penetration probability of the human body model after redirection is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than a similarity threshold, it is determined that the redirected model has not converged;

[0015] If the penetration probability of the human body model after redirection is not greater than a preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than a similarity threshold, it is determined that the redirected model has converged.

[0016] Further, the redirection model includes: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module;

[0017] The step of inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix comprises:

[0018] Inputting the original rotation information matrix and the skeleton information matrix into a redirection model;

[0019] The skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix;

[0020] The rotation information encoding module determines a rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix;

[0021] The rotation information decoding module determines a redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

[0022] Furthermore, it also includes: refining the redirection rotation information matrix;

[0023] Before judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix, the method includes:

[0024] Based on the refined redirection rotation information matrix, determining the refined three-dimensional mesh information of the human body model after redirection;

[0025] Determine a first loss of the redirected human body model according to the refined redirection rotation information matrix and the redirection rotation information matrix; and determine a second loss of the redirected human body model according to the refined three-dimensional mesh information and the three-dimensional mesh information of the redirected human body model;

[0026] The method of judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix includes:

[0027] A weighted sum is performed on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain a loss value of the redirected human body model;

[0028] Determining whether the loss value is greater than a preset loss threshold;

[0029] If so, it is determined that the redirected model has not converged; if not, it is determined that the redirected model has converged.

[0030] Furthermore, determining the three-dimensional grid information of the human body model after redirection based on the redirection rotation information matrix includes:

[0031] The redirected rotation information matrix is ​​input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the redirected human body model.

[0032] In order to achieve the above-mentioned purpose, an embodiment of the present invention further provides a model penetration correction device, comprising:

[0033] An information acquisition module, used to acquire an original rotation information matrix and a skeleton information matrix of an original human body model; the original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in T-Pose (T-letter posture);

[0034] A matrix determination module, used for inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix;

[0035] A mesh information determination module, used to determine the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix;

[0036] A penetration probability calculation module is used to calculate the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and determine the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0037] A convergence judgment module is used to judge whether the redirected model converges based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0038] A parameter adjustment module, used for adjusting the parameters of the redirection model if the redirection model has not converged, and returning to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix;

[0039] The redirection module is used to determine the redirection rotation information matrix representation if the redirection model has converged: the rotation information of the redirected human body model corresponding to the original human body model.

[0040] Furthermore, the convergence judgment module is specifically used to determine that the redirected model has not converged if the penetration probability of the human body model after redirection is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than the similarity threshold; if the penetration probability of the human body model after redirection is not greater than the preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than the similarity threshold, then determine that the redirected model has converged.

[0041] Further, the redirection model includes: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module;

[0042] The matrix determination module is specifically used to input the original rotation information matrix and the skeleton information matrix into the redirection model; the skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix; the rotation information encoding module determines the rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix; the rotation information decoding module determines the redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

[0043] Furthermore, it also includes: refining the redirection rotation information matrix;

[0044] Before judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix, the method includes:

[0045] A loss determination module is used to determine the refined three-dimensional mesh information of the redirected human body model based on the refined redirected rotation information matrix; determine the first loss of the redirected human body model according to the refined redirected rotation information matrix and the redirected rotation information matrix; and determine the second loss of the redirected human body model according to the refined three-dimensional mesh information and the three-dimensional mesh information of the redirected human body model;

[0046] The convergence judgment module is specifically used to perform weighted summation on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain the loss value of the redirected human body model; judge whether the loss value is greater than a preset loss threshold; if so, determine that the redirected model has not converged; if not, determine that the redirected model has converged.

[0047] Furthermore, determining the three-dimensional grid information of the human body model after redirection based on the redirection rotation information matrix includes:

[0048] The redirected rotation information matrix is ​​input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the redirected human body model.

[0049] In order to achieve the above object, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0050] Memory, used to store computer programs;

[0051] The processor is used to implement any of the above-mentioned model penetration correction method steps when executing the program stored in the memory.

[0052] In order to achieve the above-mentioned purpose, an embodiment of the present invention provides a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned steps of the model penetration correction method is implemented.

[0053] In order to achieve the above-mentioned purpose, an embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned steps of the model penetration correction method.

[0054] The method provided by the embodiment of the present invention is adopted to obtain the original rotation information matrix and the skeleton information matrix of the original human body model; the original rotation information matrix and the skeleton information matrix are input into the redirection model to obtain the redirection rotation information matrix; based on the redirection rotation information matrix, the three-dimensional mesh information of the redirected human body model is determined; based on the three-dimensional mesh information of the redirected human body model, the penetration probability of the redirected human body model is calculated; and, the similarity between the redirection rotation information matrix and the original rotation information matrix is ​​determined; based on the penetration probability of the redirected human body model and the similarity between the redirection rotation information matrix and the original rotation information matrix, it is judged whether the redirection model converges; if the redirection model does not converge, the parameters of the redirection model are adjusted, and the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix is ​​returned; if the redirection model has converged, it is determined that the redirection rotation information matrix represents: the rotation information of the redirected human body model corresponding to the original human body model. That is, in the process of redirecting the original model, the penetration probability of the redirected human body model is calculated, and the redirected human body model that penetrates the model is adjusted in time. That is, the mold penetration repair of the redirected human body model realized by the redirected model no longer needs to be manually repaired. Therefore, the manual consumption of mold penetration repair of the redirected human body model is reduced, and the mold penetration repair efficiency is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0056] Figure 1 This is a schematic diagram of the human body model;

[0057] Figure 2 A flow chart of a model penetration correction method provided by an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the human body model skeleton;

[0059] Figure 4 A schematic diagram of feature extraction provided by an embodiment of the present invention;

[0060] Figure 5 Another flow chart of the model penetration correction method provided by the embodiment of the present invention;

[0061] Figure 6 A schematic diagram of a model penetration correction method provided by an embodiment of the present invention;

[0062] Figure 7 A structural diagram of a model penetration correction device provided by an embodiment of the present invention;

[0063] Figure 8 Another structural diagram of the model penetration correction device provided by an embodiment of the present invention;

[0064] Fig. 9 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0066] The existing manual repair of model penetration frame by frame consumes huge labor and takes a long time, and the repair efficiency is low. In order to reduce the labor consumption of model penetration repair of the redirected human body model and improve the model penetration repair efficiency, the embodiments of the present invention provide a model penetration correction method, device, electronic device and storage medium.

[0067] See also Figure 2 , Figure 2 A process for model penetration correction method, including:

[0068] Step 201, obtaining the original rotation information matrix and skeleton information matrix of the original human body model; the original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in T-Pose (T letter posture).

[0069] In an embodiment of the present invention, the skeleton information of the original human body model includes: the relative displacement of each driving joint point of the original human body model; the original human body model includes a root joint point and multiple driving joint points; the relative displacement of each driving joint point represents: the displacement information of the driving joint point relative to the root joint point.

[0070] For an example, see Figure 3 , Figure 3is a schematic diagram of the human body model skeleton. The human body model may include multiple human body joints. The human body model T-Pose schematic diagram shows multiple human body joints: 0-hips (hip joints), 1-spine_virtual (spine virtual joints), 1-leftupleg_virtual (left upper limb virtual joints), 1-rightupleg_virtual (right upper limb virtual joints), 2-spine (spine joints), 2-leftupleg (left upper limb joints), 2-rightupleg (right upper limb joints), 3-spine1 (spine joints 1), 3-leftleg (left limb joints), 3-rightleg (right limb joints), 4-spine_split (spina bifida joints), 4-leftfoot (left foot joints), 4-rightfoot (right foot joints), 5-spine2 (spine joints 2), 5-lefttoebase (left sole joints), 5-righttoebase (right sole joints), 6-neck_virtual ( : The following are the joint points of the keyboard: 1-neck, 2-lefttoe_end, 3-righttoe_end, 4-leftshoulder_virtual, 5-rightshoulder_virtual, 6-neck, 7-leftshoulder, 7-rightshoulder, 8-head, 8-leftshoulder_split, 8-rightshoulder_split, 9-headtop_end, 9-leftarm, 9-rightarm, 10-leftforearm, 10-rightforearm, 11-lefthand, 11-righthand. The designated joints may include: left hand joints, left forearm joints, left shoulder virtual joints, right hand joints, right forearm joints, right shoulder virtual joints, left foot joints, left limb joints, left upper limb virtual joints, right foot joints, right limb joints, and right upper limb virtual joints. 0-hips (hip joints) may be used as the root joint in the original human body model, and all joints other than 0-hips (hip joints) may be used as driving joints in the original human body model.

[0071] Each driving joint point may include one or more adjacent driving joint points. For example, Figure 3 The adjacent driving joint points of the middle driving joint point 5-spine2 include: driving joint point 6-leftshoulder_virtual, driving joint point 6-rightshoulder_virtual, driving joint point 6-neck_virtual and driving joint point 4-spine_split.

[0072] In the embodiment of the present invention, the relative displacement of each driving joint point of the original human body model is the displacement information of the driving joint point relative to the root joint point. The relative displacement of each driving joint point can be calculated by the coordinates of each driving joint point and the coordinates of the root node.

[0073] In an embodiment of the present invention, the original rotation information matrix of multiple frames of the original human body model can be obtained, and each frame of the original rotation information matrix includes: the original rotation information of each driving joint point of the original human body model in the frame. The original rotation information of each driving joint point in the frame can represent: the rotation angle information of the driving joint point of the original human body model in the frame relative to each joint point of the human body model in T-Pose (T letter posture).

[0074] In the embodiment of the present invention, the Euler angle of each driving joint point of the original human body model in each frame can be converted into a quaternion, and the converted quaternion is used to represent the original rotation information of each driving joint point. Specifically, the original rotation information of each driving joint point in each frame can be represented by a quaternion:

[0075]

[0076] in, Represents the original rotation information of the i-th driving joint point in the t-th frame motion information. a, b, c and d are all constant parameters, and i, j and k are the basis of the quaternion.

[0077] Step 202: Input the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix.

[0078] In the embodiment of the present invention, the skeleton encoding model, rotation encoding model and decoding model corresponding to each driving joint point in the original human body model can be determined in advance. Figure 4 For each driving joint point in the original human body model, the skeleton information S of the driving joint point can be T Input the skeleton encoding model Encoder S corresponding to the driving joint point T In the skeleton encoding model Encoder S T The skeleton information S of the driving joint pointT Processing is performed to obtain the skeleton information feature VS of the driving joint point T ; Among them, the skeleton information S of the driving joint point T Including: the relative displacement of the driving joint point and the relative displacement of each adjacent driving joint point of the driving joint point; further, the original rotation information Q corresponding to the driving joint point and the skeleton information feature VS of the driving joint point can be T Input the rotation encoding model Encoder Q corresponding to the driving joint point, the rotation encoding model Encoder Q corresponds to the rotation information Q of the driving joint point and the skeleton information feature VS of the driving joint point T Processing is performed to obtain the rotation information feature V of the driving joint point, wherein the rotation information Q corresponding to the driving joint point includes: the original rotation information of the driving joint point and the original rotation information of each adjacent driving joint point of the driving joint point; further, the rotation information feature V of the driving joint point and the skeleton information feature VS of the driving joint point can be T The decoded rotation information Q' of the driving joint point is input into the decoding model Decoder Q corresponding to the driving joint point for processing to obtain the decoded rotation information Q' of the driving joint point; further, the error between the decoded rotation information Q' of the driving joint point and the original rotation information of the driving joint point can be calculated. If the error is less than the preset error threshold, the skeleton encoding model Encoder S corresponding to the current driving joint point A can be T Output VS T As the skeleton information feature of the driving joint point, the current rotation encoding model Encoder Q output V corresponding to the driving joint point is used as the rotation information feature of the driving joint point; if the error is not less than the preset error threshold, the skeleton encoding model Encoder S corresponding to the driving joint point can be T , the encoder model Encoder Q and the decoding model Decoder Q are rotated to adjust the model parameters until the error is less than the preset error threshold, and the Encoder S after the adjustment is T , Encoder Q and Decoder Q serve as the skeleton encoding model, rotation encoding model and decoding model corresponding to the driving joint point. The preset error threshold can be set according to actual application and is not specifically limited.

[0079] In an embodiment of the present invention, the redirection model may include: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module; wherein the skeleton information encoding module may include a skeleton encoding model of each driving joint point of the original human body model, the rotation information encoding module may include a rotation encoding model of each driving joint point of the original human body model, and the rotation information decoding module may include a decoding model of each driving joint point of the original human body model.

[0080] Specifically, in an embodiment of the present invention, inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix may include steps A1 to A4:

[0081] Step A1: Input the original rotation information matrix and skeleton information matrix into the retargeting model.

[0082] Step A2: The skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix.

[0083] The skeleton feature matrix of each driving joint point can be determined according to the skeleton information of each driving joint point in the original human body model and the skeleton coding model corresponding to each driving joint point in the skeleton information coding module.

[0084] Step A3: The rotation information encoding module determines a rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix.

[0085] The rotation feature matrix of each driving joint point can be determined according to the original rotation information of each driving joint point in the original human body model and the rotation encoding model corresponding to each driving joint point in the rotation information encoding module.

[0086] Step A4: The rotation information decoding module determines the redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

[0087] The decoded rotation information of each driving joint point can be determined based on the rotation feature matrix and skeleton feature matrix of each driving joint point and the decoding model corresponding to each driving joint point in the rotation information decoding module, and the matrix composed of the decoded rotation information of each driving joint point can be used as the redirected rotation information matrix.

[0088] Step 203: Determine the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix.

[0089] In the embodiment of the present invention, the redirected rotation information matrix can be input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the human body model after redirection.

[0090] Step 204, based on the three-dimensional mesh information of the redirected human body model, calculate the penetration probability of the redirected human body model; and determine the similarity between the redirected rotation information matrix and the original rotation information matrix.

[0091] In the embodiment of the present invention, the penetration probability of the human body model after redirection can be calculated using the following formula:

[0092]

[0093] Where P is the penetration probability of the human body model, A represents the set of triangles corresponding to the first type of driving joints of the human body model, and B represents the set of triangles corresponding to the second type of driving joints of the human body model; a is an element in set A, that is, a represents the triangle corresponding to the driving joint in set A; b is an element in set B, that is, b represents the triangle corresponding to the driving joint, where a∈A, b∈B; P a is the centroid coordinate of the triangle corresponding to the first type of driving joint point a, P b is the centroid coordinate of the triangle corresponding to the second type of driving joint point b; d(P a , P b ) is the distance between the centroid of the triangle corresponding to the first type of driving joint point a and the centroid of the triangle corresponding to the second type of driving joint point b; ω a-b P a and P b The weight between .

[0094] In the embodiment of the present invention, the penetration probability of the human body model after redirection may be determined by using any existing publicly available method for calculating the penetration probability, which will not be described in detail herein.

[0095] In the embodiment of the present invention, the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​used to ensure that the movement of the redirected human body model is close to the movement of the original human body model. To calculate the similarity between the redirected rotation information matrix and the original rotation information matrix, any existing motion similarity calculation method can be used, which is not limited here.

[0096] Step 205 , based on the penetration probability of the human body model after redirection and the similarity between the redirected rotation information matrix and the original rotation information matrix, it is determined whether the redirected model has converged.

[0097] In the embodiment of the present invention, judging whether the redirection model has converged may include the following steps B1-B2:

[0098] Step B1: if the penetration probability of the redirected human body model is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than a similarity threshold, it is determined that the redirected model has not converged;

[0099] Step B2: If the penetration probability of the redirected human body model is not greater than the preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than the similarity threshold, it is determined that the redirected model has converged.

[0100] Among them, the similarity threshold and the preset probability threshold can be adaptively set according to the actual application situation and are not limited here.

[0101] In this step, if the penetration probability of the redirected human body model is greater than a preset probability threshold, it means that a penetration phenomenon has occurred in the redirected human body model.

[0102] Step 206 : If the redirection model has not converged, adjust the parameters of the redirection model and return to step 202 .

[0103] Specifically, the parameters of the fully connected layer of the skeleton encoding model of each driving joint point of the original human body model included in the skeleton information encoding module in the redirection model can be adjusted, as can the parameters of the fully connected layer of the rotation encoding model of each driving joint point of the original human body model included in the rotation information encoding module, and the parameters of the fully connected layer of the decoding model of each driving joint point of the original human body model included in the rotation information decoding module can be adjusted.

[0104] Step 207: If the redirected model has converged, determine the redirected rotation information matrix to represent: the rotation information of the redirected human body model corresponding to the original human body model.

[0105] The method provided by the embodiment of the present invention is adopted to obtain the original rotation information matrix and the skeleton information matrix of the original human body model; the original rotation information matrix and the skeleton information matrix are input into the redirection model to obtain the redirection rotation information matrix; based on the redirection rotation information matrix, the three-dimensional mesh information of the redirected human body model is determined; based on the three-dimensional mesh information of the redirected human body model, the penetration probability of the redirected human body model is calculated; and, the similarity between the redirection rotation information matrix and the original rotation information matrix is ​​determined; based on the penetration probability of the redirected human body model and the similarity between the redirection rotation information matrix and the original rotation information matrix, it is judged whether the redirection model converges; if the redirection model does not converge, the parameters of the redirection model are adjusted, and the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix is ​​returned; if the redirection model has converged, it is determined that the redirection rotation information matrix represents: the rotation information of the redirected human body model corresponding to the original human body model. That is, in the process of redirecting the original model, the penetration probability of the redirected human body model is calculated, and the redirected human body model that penetrates the model is adjusted in time. That is, the mold penetration repair of the redirected human body model realized by the redirected model no longer needs to be manually repaired. Therefore, the manual consumption of mold penetration repair of the redirected human body model is reduced, and the mold penetration repair efficiency is also improved.

[0106] See also Figure 5 , Figure 5 Another process for the model penetration correction method includes:

[0107] Step 501 to step 504 are the same as step 201 to step 204, and will not be repeated here.

[0108] Step 505: Determine the refined three-dimensional mesh information of the human body model after the redirection based on the refined redirection rotation information matrix.

[0109] In the embodiment of the present invention, one or more frames of the multiple frames of original rotation information matrices may have corresponding refined redirected rotation information matrices. For example, the original rotation information matrix Q∈R (T×J×4) , refine the redirection rotation information matrix Q T ∈R (T ×J×4) . Where T represents the number of frames, and J represents the number of driving joints. In addition, only the original rotation information of some frames corresponds to the refined redirected rotation information Q T ∈R (T×J×4)For frames without refined redirection rotation information, a J×4 zero matrix can be used to replace the refined redirection rotation information matrix of this frame. The refined redirection rotation information matrix of each frame is the refined redirection rotation information matrix of the human body model after the human body model of the frame is manually repaired through the model.

[0110] In the embodiment of the present invention, the refined redirection rotation information matrix can be input into SMPL (skinned multi-person linear model) to obtain the refined three-dimensional mesh information of the human body model after redirection.

[0111] Step 506, determining a first loss of the redirected human body model according to the refined redirected rotation information matrix and the redirected rotation information matrix; and determining a second loss of the redirected human body model according to the refined 3D mesh information and the 3D mesh information of the redirected human body model.

[0112] In an embodiment of the present invention, the second loss is to calculate the loss value between the refined three-dimensional mesh information of a frame and the three-dimensional mesh information of the human body model after redirection when there is a frame of refined redirection rotation information. Specifically, a common loss function of deep learning can be used, such as the Loss1 loss function or the Loss2 loss function.

[0113] In an embodiment of the present invention, the first loss is to calculate the error between the refined redirected rotation information matrix and the redirected rotation information matrix under the refined redirected rotation information matrix. Specifically, a common loss function of deep learning can be used, such as the Loss1 loss function or the Loss2 loss function.

[0114] Step 507, performing weighted summation on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain the loss value of the redirected human body model.

[0115] Specifically, the loss value of the human body model after redirection can be calculated according to the following formula:

[0116] Loss = λ1L1+λ2L1+λ a La+λ b Lb

[0117] Among them, Loss represents the loss value of the redirected human body model, L1, L2, La, and Lb represent the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix, respectively. aand λ b They respectively represent the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the coefficients of the similarity between the redirected rotation information matrix and the original rotation information matrix.

[0118] Step 508, determine whether the loss value is greater than a preset loss threshold, if the determination result is yes, execute step 509, if the determination result is no, execute step 510.

[0119] The preset loss threshold can be specifically set according to the actual application situation and is not limited here.

[0120] Step 509 , adjusting the parameters of the redirection model, and returning to step 502 .

[0121] Step 510: Determine the redirected rotation information matrix representation: the rotation information of the redirected human body model corresponding to the original human body model.

[0122] By adopting the method provided by the embodiment of the present invention, in the process of redirecting the original model, the penetration probability of the redirected human body model is calculated, and the redirected human body model with penetration is adjusted in time. That is, the penetration repair of the redirected human body model implemented by the redirected model no longer needs to be manually performed on the redirected human body model with penetration. Therefore, the labor consumption of the penetration repair of the redirected human body model is reduced, and the penetration repair efficiency is also improved.

[0123] For example, Figure 6 This is a schematic diagram of the model penetration correction method, see Figure 6 , the original rotation information matrix and the skeleton information matrix are input into the redirection model, and the redirected rotation information matrix can be obtained. The redirected three-dimensional mesh of the human body model can be determined based on the redirected rotation information matrix and the SMPL model. The penetration probability of the redirected human body model can be calculated based on the redirected three-dimensional mesh. At the same time, the similarity between the redirected rotation information matrix and the original rotation information matrix can be determined. If there is a refined redirection rotation information matrix, the refined three-dimensional mesh information of the redirected human body model can be determined based on the refined redirection rotation information matrix; further, the first loss of the redirected human body model can be determined based on the refined redirection rotation information matrix and the redirection rotation information matrix, and the second loss of the redirected human body model can be determined based on the refined three-dimensional mesh information and the three-dimensional mesh information of the redirected human body model.

[0124] Based on the same inventive concept, according to the model penetration correction method provided by the above embodiment of the present invention, correspondingly, another embodiment of the present invention also provides a model penetration correction device, whose structural schematic diagram is shown as follows: Figure 7As shown, specifically including:

[0125] The information acquisition module 701 is used to acquire the original rotation information matrix and the skeleton information matrix of the original human body model; the original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in T-Pose (T letter posture);

[0126] A matrix determination module 702, configured to input the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix;

[0127] A mesh information determination module 703, used to determine the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix;

[0128] A penetration probability calculation module 704 is used to calculate the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and determine the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0129] A convergence judgment module 705 is used to judge whether the redirected model is converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0130] A parameter adjustment module 706 is used to adjust the parameters of the redirection model if the redirection model has not converged, and return to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix;

[0131] The redirection module 707 is used to determine the redirection rotation information matrix representation: the rotation information of the redirected human body model corresponding to the original human body model if the redirection model has converged.

[0132] The device provided by the embodiment of the present invention is used to obtain the original rotation information matrix and the skeleton information matrix of the original human body model; the original rotation information matrix and the skeleton information matrix are input into the redirection model to obtain the redirection rotation information matrix; based on the redirection rotation information matrix, the three-dimensional mesh information of the redirected human body model is determined; based on the three-dimensional mesh information of the redirected human body model, the penetration probability of the redirected human body model is calculated; and, the similarity between the redirection rotation information matrix and the original rotation information matrix is ​​determined; based on the penetration probability of the redirected human body model and the similarity between the redirection rotation information matrix and the original rotation information matrix, it is determined whether the redirection model converges; if the redirection model does not converge, the parameters of the redirection model are adjusted, and the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix is ​​returned; if the redirection model has converged, it is determined that the redirection rotation information matrix represents: the rotation information of the redirected human body model corresponding to the original human body model. That is, in the process of redirecting the original model, the penetration probability of the redirected human body model is calculated, and the redirected human body model that penetrates the model is adjusted in time. That is, the mold penetration repair of the redirected human body model realized by the redirected model no longer needs to be manually repaired. Therefore, the manual consumption of mold penetration repair of the redirected human body model is reduced, and the mold penetration repair efficiency is also improved.

[0133] Furthermore, the convergence judgment module 705 is specifically used to determine that the redirected model has not converged if the penetration probability of the human body model after redirection is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than the similarity threshold; if the penetration probability of the human body model after redirection is not greater than the preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than the similarity threshold, then determine that the redirected model has converged.

[0134] Furthermore, the device further comprises: the redirection model comprises: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module;

[0135] The matrix determination module 702 is specifically used to input the original rotation information matrix and the skeleton information matrix into the redirection model; the skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix; the rotation information encoding module determines the rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix; the rotation information decoding module determines the redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

[0136] For further information, see Figure 8, the device further comprises: refining the redirection rotation information matrix;

[0137] The loss determination module 801 is used to determine the refined 3D mesh information of the redirected human body model based on the refined redirected rotation information matrix; determine the first loss of the redirected human body model according to the refined redirected rotation information matrix and the redirected rotation information matrix; and determine the second loss of the redirected human body model according to the refined 3D mesh information and the 3D mesh information of the redirected human body model;

[0138] The convergence judgment module 705 is specifically used to perform weighted summation on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain the loss value of the redirected human body model; judge whether the loss value is greater than a preset loss threshold; if so, determine that the redirected model has not converged; if not, determine that the redirected model has converged.

[0139] Furthermore, determining the three-dimensional grid information of the human body model after redirection based on the redirection rotation information matrix includes:

[0140] The redirected rotation information matrix is ​​input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the redirected human body model.

[0141] By using the device provided by the embodiment of the present invention, in the process of redirecting the original model, the penetration probability of the redirected human body model is calculated, and the redirected human body model with penetration is adjusted in time. That is, the penetration repair of the redirected human body model implemented by the redirected model no longer needs to be manually repaired. Therefore, the labor consumption of the penetration repair of the redirected human body model is reduced, and the penetration repair efficiency is also improved.

[0142] The embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0143] Memory 903, used for storing computer programs;

[0144] The processor 901 is used to execute the program stored in the memory 903 to implement the following steps:

[0145] Acquire an original rotation information matrix and a skeleton information matrix of the original human body model; the original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in T-Pose (T-letter posture);

[0146] Inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix;

[0147] Based on the redirection rotation information matrix, determine the three-dimensional mesh information of the human body model after redirection;

[0148] Calculating the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and determining the similarity between the redirected rotation information matrix and the original rotation information matrix;

[0149] Based on the penetration probability of the human body model after redirection and the similarity between the redirected rotation information matrix and the original rotation information matrix, determine whether the redirected model converges;

[0150] If the redirection model has not converged, the parameters of the redirection model are adjusted, and the process returns to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix;

[0151] If the redirected model has converged, the redirected rotation information matrix is ​​determined to represent: the rotation information of the redirected human body model corresponding to the original human body model.

[0152] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0153] The communication interface is used for communication between the above terminal and other devices.

[0154] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0156] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the model penetration correction method described in any of the above embodiments is implemented.

[0157] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the model penetration correction method described in any one of the above embodiments.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0159] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0160] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device and storage medium, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A model penetration correction method, characterized in that: include: Obtain the original rotation information matrix and skeleton information matrix of the original human body model; The original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in the T-pose T-Pose; Inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix; Based on the redirection rotation information matrix, determine the three-dimensional mesh information of the human body model after redirection; Based on the three-dimensional mesh information of the redirected human body model, the penetration probability of the redirected human body model is calculated; and, determining a similarity between the redirected rotation information matrix and the original rotation information matrix; If the penetration probability of the human body model after redirection is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than a similarity threshold, it is determined that the redirected model has not converged, and the parameters of the redirected model are adjusted, and the process returns to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirected model to obtain the redirected rotation information matrix; If the penetration probability of the redirected human body model is not greater than the preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than the similarity threshold, it is determined that the redirected model has converged, and the redirected rotation information matrix is ​​determined to represent: the rotation information of the redirected human body model corresponding to the original human body model.

2. The method according to claim 1, characterized in that The redirection model includes: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module; The step of inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix comprises: Inputting the original rotation information matrix and the skeleton information matrix into a redirection model; The skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix; The rotation information encoding module determines a rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix; The rotation information decoding module determines a redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

3. The method according to claim 1, characterized in that Also includes: Refine the reorientation rotation information matrix; Before judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix, the method includes: Based on the refined redirection rotation information matrix, determining the refined three-dimensional mesh information of the human body model after redirection; Determine a first loss of the redirected human body model according to the refined redirected rotation information matrix and the redirected rotation information matrix; and determine a second loss of the redirected human body model according to the refined three-dimensional mesh information and the three-dimensional mesh information of the redirected human body model; The method of judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix includes: A weighted sum is performed on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain a loss value of the redirected human body model; Determining whether the loss value is greater than a preset loss threshold; If so, it is determined that the redirected model has not converged; if not, it is determined that the redirected model has converged.

4. The method according to claim 1, characterized in that The step of determining the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix includes: The redirected rotation information matrix is ​​input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the redirected human body model.

5. A model penetration correction device, characterized in that: include: An information acquisition module, used to acquire the original rotation information matrix and skeleton information matrix of the original human body model; The original rotation information represents the rotation angle information of each joint point of the original human body model relative to each joint point of the human body model in the T-pose T-Pose; A matrix determination module, used for inputting the original rotation information matrix and the skeleton information matrix into a redirection model to obtain a redirection rotation information matrix; A mesh information determination module, used to determine the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix; A penetration probability calculation module, used for calculating the penetration probability of the redirected human body model based on the three-dimensional mesh information of the redirected human body model; and, determining a similarity between the redirected rotation information matrix and the original rotation information matrix; A convergence judgment module, used to determine that the redirected model has not converged if the penetration probability of the human body model after redirection is greater than a preset probability threshold, and / or the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​greater than a similarity threshold; If the penetration probability of the human body model after redirection is not greater than a preset probability threshold, and the similarity between the redirected rotation information matrix and the original rotation information matrix is ​​not greater than a similarity threshold, it is determined that the redirected model has converged; A parameter adjustment module, used for adjusting the parameters of the redirection model if the redirection model has not converged, and returning to the step of inputting the original rotation information matrix and the skeleton information matrix into the redirection model to obtain the redirection rotation information matrix; The redirection module is used to determine the redirection rotation information matrix representation if the redirection model has converged: the rotation information of the redirected human body model corresponding to the original human body model.

6. The device according to claim 5, characterized in that The redirection model includes: a skeleton information encoding module, a rotation information encoding module and a rotation information decoding module; The matrix determination module is specifically used to input the original rotation information matrix and the skeleton information matrix into the redirection model; the skeleton information encoding module processes the skeleton information matrix to obtain a skeleton feature matrix; the rotation information encoding module determines the rotation feature matrix based on the original rotation information matrix and the skeleton feature matrix; the rotation information decoding module determines the redirection rotation information matrix based on the rotation feature matrix and the skeleton feature matrix.

7. The device according to claim 5, characterized in that Also includes: Refine the reorientation rotation information matrix; Before judging whether the redirected model has converged based on the penetration probability of the redirected human body model and the similarity between the redirected rotation information matrix and the original rotation information matrix, the method includes: A loss determination module is used to determine the refined three-dimensional mesh information of the redirected human body model based on the refined redirected rotation information matrix; determine the first loss of the redirected human body model according to the refined redirected rotation information matrix and the redirected rotation information matrix; and determine the second loss of the redirected human body model according to the refined three-dimensional mesh information and the three-dimensional mesh information of the redirected human body model; The convergence judgment module is specifically used to perform weighted summation on the first loss of the redirected human body model, the second loss of the redirected human body model, the penetration probability of the redirected human body model, and the similarity between the redirected rotation information matrix and the original rotation information matrix to obtain the loss value of the redirected human body model; judge whether the loss value is greater than a preset loss threshold; if so, determine that the redirected model has not converged; if not, determine that the redirected model has converged.

8. The device according to claim 5, characterized in that The step of determining the three-dimensional mesh information of the human body model after redirection based on the redirection rotation information matrix includes: The redirected rotation information matrix is ​​input into SMPL (skinned multi-person linear model) to obtain the three-dimensional mesh information of the redirected human body model.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images