Action posture information completion method, electronic device and computer readable storage medium

Abstract

The motion posture information completion method, electronic device and computer-readable storage medium provided in the embodiments of the present application relate to the field of artificial intelligence technology. The relative position relationship of invisible key points is predicted based on the relative position relationship of visible key points in multiple motion posture frames in the time sequence in the motion posture sequence information. Compared with predicting invisible key points based on a single motion posture frame or a single image, the problem that the relative position relationship of invisible key points cannot be predicted due to the lack of relative position change relationship between key points in time sequence can be avoided. By using a trained motion posture completion model based on the change of the relative position relationship of the key points in time sequence to predict the relative position relationship of the invisible key points, the second motion posture frame sequence information after the motion posture information is completed can be made consistent with the actual motion posture of the target object.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for completing motion posture information, an electronic device, and a computer-readable storage medium. Background Art

[0002] Posture recognition technology is an important technical branch of artificial intelligence. Predicting the action posture of the target object through the key points (e.g., joint center points) of the target object is an important implementation method of posture recognition. However, the key points of the target object may be invisible. When performing back-end applications (e.g., virtual object posture driving), the relative position information of the invisible key points needs to be completed first. Based on this method, the problem of large difference between the action posture information after completion and the actual action posture of the target object may occur due to the poor completion effect in the completion process, which ultimately leads to a large difference between the action posture of the virtual object and the actual action posture of the target object. Therefore, how to reduce the difference between the action posture information after completion and the actual action posture of the target object is a technical problem that those skilled in the art are in urgent need of solving. Summary of the invention

[0003] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the present application is to provide a method for completing action posture information, an electronic device and a computer-readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides a method for completing action posture information, the method comprising:

[0005] Acquire first motion gesture frame sequence information of the target object, wherein the first motion gesture frame sequence information includes visibility information of each key point in the target object in each motion gesture frame and relative position information of visible key points in each key point in each motion gesture frame, and there is at least one invisible key point in each key point, wherein the relative position information of the key point refers to information used to confirm the relative position of the key point with other key points, and the relative position information of the key point is a three-dimensional axis angle;

[0006] The first motion posture frame sequence information is input into the trained motion posture completion model for prediction, and the relative position information of the invisible key points in the first motion posture frame sequence information is completed to obtain the second motion posture frame sequence information, wherein the trained motion posture completion model predicts the relative position information of the invisible key points based on the relative position information of the visible key points in each of the motion posture frames in time sequence.

[0007] In the above scheme, the first action posture sequence information of the target object is input into the trained action posture completion model for prediction, and the relative position information of the invisible key points in the first action posture frame sequence information is obtained. The relative position relationship of the invisible key points is predicted based on the relative position relationship of the visible key points in multiple action posture frames in the time sequence in the action posture sequence information. Compared with the prediction of the invisible key points based on a single action posture frame or a single image, the problem that the relative position relationship of the invisible key points cannot be predicted due to the lack of relative position change relationship between the key points in time sequence can be avoided. By using the trained action posture completion model to predict the relative position relationship of the invisible key points based on the change of the relative position relationship of the key points in time sequence, the second action posture frame sequence information after the action posture information is completed can be made consistent with the actual action posture of the target object.

[0008] In one possible implementation, the action posture completion model includes a generator, the generator includes an encoder and a decoder, the encoder and the decoder are composed of a multi-layer fully connected network with the same number of layers, the encoder is connected to the decoder, the encoder performs dimensionality reduction processing on the data input into the generator, and the decoder performs dimensionality increase processing on the data output by the encoder.

[0009] In a possible implementation, the dimension of the output data of the last fully connected network layer of the encoder includes one dimension from 512 dimensions to 32 dimensions.

[0010] In a possible implementation, the method further includes a step of training the action posture completion model, which includes:

[0011] Acquire third action posture frame sequence information of the target object;

[0012] Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information;

[0013] The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information;

[0014] Based on the third motion posture frame sequence information and the completed motion posture frame sequence information, determine whether the motion posture completion model has converged. If it has not converged, adjust the network parameters of the generator and return to the step of obtaining the third motion posture frame sequence information of the target object until the motion posture completion model converges to obtain a trained motion posture completion model.

[0015] In a possible implementation, the step of obtaining the action gesture frame sequence sample information based on the invisible probability of the key point in the target object and the third action gesture frame sequence information includes:

[0016] Determining a key point mask corresponding to a key point in each action posture frame in the third action posture frame sequence information based on the invisible probability of different key points in the target object, wherein the key point mask corresponding to the visible key point is set to a first preset value, and the key point mask corresponding to the invisible key point is set to a second preset value;

[0017] The three-dimensional axis angle of the key point in the third action posture frame sequence information and the corresponding key point mask are superimposed to obtain the action posture frame sequence sample information, wherein when the key point is not visible, the three-dimensional axis angle corresponding to the key point is set to the second preset value.

[0018] In a possible implementation, the generator includes an encoder and a decoder, and the step of predicting the relative position information of invisible key points in the action posture frame sequence sample information by the generator to obtain the completed action posture frame sequence information includes:

[0019] Inputting the action posture frame sequence sample information into the encoder for feature extraction;

[0020] Inputting the extracted features into the decoder to perform feature reconstruction to obtain reconstructed features;

[0021] The relative position information of the visible key points in the third action posture frame sequence information is used to replace the relative position information of the corresponding visible key points in the reconstructed feature to obtain an updated reconstructed feature, and the updated reconstructed feature is used as the completed action posture frame sequence information.

[0022] In a possible implementation, the step of judging whether the action posture completion model has converged based on the third action posture frame sequence information and the completed action posture frame sequence information includes:

[0023] Calculating a first loss function value of a generator based on the third action posture frame sequence information and the completed action posture frame sequence information, and judging whether the action posture completion model converges based on the first loss function value of the generator;

[0024] The calculation formula of the first loss function value Lg of the generator is:

[0025]

[0026] Wherein, T is the number of time series frames, t is the t-th frame in the time series frame, G is the number of key points of the target object, g represents the g-th key point, M(t,g) is the mask of the g-th key point in the t-th frame, M(t,g) is the first preset value when the key point is visible, and M(t,g) is the second preset value when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, It is the predicted relative position information of the key points in the completed action posture frame sequence information.

[0027] In one possible implementation, the action posture completion model includes an adversarial network consisting of a generator and a discriminator, the generator includes an encoder and a decoder, the encoder and the decoder are composed of multi-layer fully connected networks with the same number of layers, the encoder is connected to the decoder, the encoder performs dimensionality reduction processing on the data input to the generator, and the decoder performs dimensionality increase processing on the data output by the encoder, the discriminator is composed of multi-layer fully connected networks, and the discriminator is connected to the generator, wherein the dimension of the output data of the last layer of the fully connected network of the encoder includes one dimension from 512 dimensions to 32 dimensions.

[0028] In a possible implementation, the method further includes a step of training the action posture completion model, which includes:

[0029] Acquire third motion posture frame sequence information and motion posture timing frame information of the target object;

[0030] Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information;

[0031] The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information;

[0032] Inputting the completed motion gesture frame sequence information and the motion gesture time sequence frame information as input information into the discriminator for training, and obtaining a discriminant probability that the input information output by the discriminator is non-completed motion gesture frame sequence information;

[0033] Calculate the second loss function value of the generator and the loss function value of the discriminator based on the first loss function value of the generator, the discrimination probability of the discriminator for the action posture time sequence frame information, and the discrimination probability of the discriminator for the completed action posture frame sequence information;

[0034] When the second loss function value of the generator and the loss function value of the discriminator do not meet the convergence condition, adjust the network parameters in the generator and the discriminator, and return to the step of obtaining the third action posture frame sequence information and action posture timing frame information of the target object until the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition.

[0035] In a possible implementation, the first loss function value Lg of the generator is calculated by the following calculation formula:

[0036]

[0037] The second loss function value LG of the generator is calculated by the following calculation formula:

[0038] L G =L g +(D(G(input))-1) 2

[0039] The calculation formula of the loss function value Ld of the discriminator is as follows:

[0040] L d =(1-D(gt)) 2 +(D(G(input))) 2

[0041] Wherein, T is the number of time series frames, t is the t-th frame in the time series frames, G is the number of key points of the target object, g represents the g-th key point, M(t, g) is the mask of the g-th key point in the t-th frame, M(t, g) is the first preset value when the key point is visible, and M(t, g) is the second preset value when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, is the predicted relative position information of the key points in the completed action posture frame sequence information, input is the action posture frame sequence sample information, G(input) is the completed action posture frame sequence information output by the generator, D(G(input)) is the discriminant probability of the completed action posture frame sequence information, gt is the action posture timing frame information, and D(gt) is the discriminant probability of the action posture timing frame information.

[0042] In a possible implementation, after the step of inputting the first action posture frame sequence information into a trained action posture completion model for training, completing the relative position information of invisible key points in the first action posture frame sequence information, and obtaining the second action posture frame sequence information, the method further includes:

[0043] The gesture action of the virtual image corresponding to the target object is driven based on the second action gesture frame sequence information.

[0044] In a second aspect, an embodiment of the present application further provides an action posture information completion device, the device comprising:

[0045] An acquisition module is used to acquire first motion gesture frame sequence information of a target object, wherein the first motion gesture frame sequence information includes visibility information of each key point in the target object in each motion gesture frame and relative position information of visible key points in each of the key points in each of the key points in each of the motion gesture frames, wherein at least one invisible key point exists in each of the key points, wherein the relative position information of the key point refers to information used to confirm the relative position of the key point with other key points, and the relative position information of the key point is a three-dimensional axis angle;

[0046] A completion module is used to input the first action posture frame sequence information into a trained action posture completion model for prediction, complete the relative position information of invisible key points in the first action posture frame sequence information, and obtain second action posture frame sequence information, wherein the trained action posture completion model predicts the relative position information of invisible key points based on the relative position information of visible key points in each of the action posture frames in time sequence.

[0047] In the third aspect, an embodiment of the present application also provides an electronic device, which includes a processor and a computer-readable storage medium, wherein the processor and the computer-readable storage medium are connected via a bus system, the computer-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the computer-readable storage medium to implement the motion posture information completion method described in the first aspect.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored, which, when executed, enable a computer to execute the motion posture information completion method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0049] Based on any one of the above aspects, the embodiments of the present application provide a method for completing motion gesture information, an electronic device, and a computer-readable storage medium. The relative position relationship of invisible key points is predicted based on the relative position relationship of visible key points in multiple motion gesture frames in the time sequence in the motion gesture sequence information. Compared with the method of predicting invisible key points based on a single motion gesture frame or a single image, the problem that the relative position relationship of invisible key points cannot be predicted due to the lack of relative position change relationship in time sequence between key points can be avoided. In addition, by using a trained motion gesture completion model to predict the relative position relationship of invisible key points based on the change of the relative position relationship of the key points in time sequence, the second motion gesture frame sequence information after the motion gesture information is completed can be made consistent with the actual motion gesture of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A schematic diagram of a flow chart of a method for completing action posture information provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a training step flow of the action posture completion model provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a model of an action posture completion model provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of another training step flow of the action posture completion model provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of another action posture completion model provided in an embodiment of the present application;

[0056] Figure 6 Another schematic diagram of a flow chart of a method for completing action posture information provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of a functional module of the motion posture information completion device provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of another functional module of the motion posture information completion device provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The present application is described in detail below in conjunction with the drawings in the specification. The specific operation method in the method embodiment can also be applied to the device embodiment or the system embodiment.

[0061] In order to facilitate understanding of the technical issues mentioned in the background technology, this embodiment takes a scene based on virtual live broadcast as an example, and takes the action posture of the corresponding virtual object driven by the key points of the detected target object (for example, the anchor) as an example. Because the arms and legs of the target object may not be visible (for example, blocked or located outside the capture range of the motion capture device), the key points corresponding to the arms and legs cannot be detected during detection. When the action posture of the corresponding virtual object is driven based on the key points, because the key points corresponding to the arms and legs are invisible, the action posture of the driven virtual object will be very different from the actual action posture of the target object. Especially in the transition stage where all the key points of the target object are visible and some of the key points of the target object are invisible, there will be screen jitter caused by excessive changes in the action posture of the virtual object, which seriously affects the live broadcast effect.

[0062] In order to solve the above technical problems, there are the following two possible solutions in the prior art.

[0063] The first possible solution is the image-driven limb solution, which predicts the rotation direction of the key points of the target object (for example, the center point of the joint) through the RGB image input by the user. Specifically, the ResNet50 network with large computational complexity can be used to extract features, combined with a recurrent neural network to predict the signal data for limb driving, and then an optimization algorithm is used to optimize the predicted signal data to determine the rotation direction of the key points of the target object.

[0064] The second possible solution is the key point driven limb solution, which predicts the coordinates of all key points (for example, the center points of joints) through the RGB image input by the user, and then obtains the rotation direction of each key point of the target object based on the coordinates of the key points.

[0065] The inventors have found that the above two possible solutions have the following shortcomings.

[0066] For the first possible solution, when part of the limbs (for example, the lower body) is invisible, since the input image is quite different from the sample image used during model training, sample images of such part of the limbs that are invisible may not be used during training, which may lead to errors in the prediction of the invisible limb parts.

[0067] For the second possible solution, when part of the limbs (for example, the lower body) is not visible, the robustness of this solution is even worse. Due to the lack of necessary connection between the visible key points and the invisible key points, the prediction effect of the invisible key points based on the visible key points is very poor. When the limbs are occluded, such as the arms, the prediction of the key points corresponding to the arms is inaccurate, which will cause the key points corresponding to the arms to shake violently, which will have a great impact on the subsequent posture prediction.

[0068] In order to overcome the above-mentioned defects, the present application embodiment provides a method for completing action posture information. Figure 1 As shown, it is a flow chart of a method for completing motion posture information provided by an embodiment of the present application. The method for completing motion posture information provided by an embodiment of the present application can be executed by an electronic device with data processing capability. Figure 1 The action posture information completion method provided in this embodiment is introduced in detail.

[0069] Step S11, obtaining first motion posture frame sequence information of the target object.

[0070] In the embodiment of the present application, the target object can be a person (e.g., a network anchor) or other animals (e.g., a panda). In the present embodiment, the target object is a person (e.g., an anchor of a live broadcast scene) as an example for explanation.

[0071] Wherein, the first motion gesture frame sequence information can be composed of information of multiple motion gesture frames that are continuous in time sequence, and the information of each motion gesture frame includes visibility information of each key point in the target object and relative position information of visible key points among each key point in the motion gesture frame. Wherein, the visibility information can include indication information of whether the key point is visible, for example, including first indication information that the key point is visible and second indication information that the key point is invisible. Specifically, the visibility information can be represented by a digital mark, for example, for a visible key point, the number 1 can be used to represent the visibility information of the key point, and for an invisible key point, the number 0 can be used to represent the visibility information of the key point. Wherein, the motion gesture frame can be obtained by a motion capture device, and the motion capture device can include an optical motion capture device and a thermal imaging motion capture device, etc. The relative position information of the key point refers to information used to confirm the relative position of the key point with other key points, wherein other key points can be adjacent key points of the key point, or a reference key point.

[0072] Among them, the key point can be the joint center point corresponding to the target object. Specifically, the center point of the main joint that can be used to drive the limb can be used as the key point. For example, when the target object is a person, the key point can include the pelvic joint center point, the left hip joint center point, the right hip joint center point, the first spinal joint center point, the left knee joint center point, the right knee joint center point, the second spinal joint center point, the left ankle joint center point, the right ankle joint center point, the third spinal joint center point, the left foot joint center point, the right foot joint center point, the neck joint center point, the left neck joint center point, the right neck joint center point, the head joint center point, the left shoulder joint center point, the right shoulder joint center point, the left elbow joint center point, the right elbow joint center point, the left wrist joint center point, the right wrist joint center point, the left hand joint center point and the right hand joint center point. It can be understood that the above is only an example of illustrating the composition of the key points. In other embodiments of the present application, the number and type of key points can be adjusted accordingly.

[0073] Step S12: input the first action posture frame sequence information into the trained action posture completion model for prediction, complete the relative position information of the invisible key points in the first action posture frame sequence information, and obtain the second action posture frame sequence information.

[0074] Specifically, the trained action posture completion model can predict the relative position information of the invisible key points according to the relative position information of the visible key points in each action posture frame that is consecutive in time sequence in the first action posture frame sequence information.

[0075] Based on the above scheme, the relative position relationship of invisible key points is predicted based on the relative position relationship of visible key points in multiple action posture frames in time sequence in the action posture sequence information. Compared with predicting invisible key points based on a single action posture frame or a single image, the problem of unpredictable relative position relationship of invisible key points due to the lack of relative position change relationship between key points in time sequence can be avoided. By predicting the relative position relationship of invisible key points based on the change of the relative position relationship of key points in time sequence using a trained action posture completion model, the second action posture frame sequence information after the action posture information is completed can be consistent with the actual action posture of the target object.

[0076] In a possible implementation of the embodiment of the present application, the method for completing the action posture information provided in the embodiment of the present application further includes the step of training an action posture completion model, wherein the action posture completion model may include a generator, please refer to Figure 2 and Figure 3 , Figure 2 shows a schematic diagram of a step flow for training an action posture completion model, Figure 3 A schematic diagram of the model structure of the action posture completion model is shown below. Figure 2 and Figure 3 A detailed introduction to the steps of training a pose completion model.

[0077] Step S131, obtaining third action posture frame sequence information of the target object.

[0078] Among them, the third action posture frame sequence information can be the action posture frame sequence information derived from the key point detection model, wherein the key point detection model can detect the relative position information of each key point of the target object, and all key points in the third action posture frame sequence information can be visible key points, that is, the key points are visible in each action posture frame sequence corresponding to the third action posture frame sequence.

[0079] Step S132, obtaining action posture frame sequence sample information based on the invisible probability of the key point in the target object and the third action posture frame sequence information.

[0080] In actual application scenarios, the invisibility probabilities of various key points may be different. Taking the live broadcast scenario as an example, as shown in Table 1, the invisibility probabilities of wrists and feet are higher than those of elbows, knees and hips. Therefore, different invisibility probabilities need to be set for key points corresponding to different joints in the obtained action posture frame sequence sample information.

[0081] joint Wrists, feet knee elbow Hip other Invisible probability 0.9 0.8 0.7 0.5 0.1

[0082] Table 1

[0083] Such a setting can enable the action posture completion model to conduct targeted relative position information training on key points with a high probability of being invisible, thereby improving the prediction accuracy of the action posture completion model for the relative position information of such invisible key points.

[0084] The action posture completion model of the embodiment of the present application is not coupled with the key point detection model for key point detection. Therefore, the action posture information completion method provided in the embodiment of the present application needs to obtain the action posture frame sequence sample information through simulation.

[0085] In an embodiment of the present application, the relative position information of the key point can be a three-dimensional axis angle, wherein the three-dimensional axis angle refers to the angle between the joint axes of the joints corresponding to the key points, which can generally be expressed as α, β, γ, etc. The method for obtaining the action posture frame sequence sample information can be as follows.

[0086] Specifically, the key point mask corresponding to the key point in each action gesture frame in the third action gesture frame sequence information can be determined based on the invisible probability of different key points in the target object, wherein the key point mask corresponding to the visible key point can be set to a first preset value (for example, 1), and the key point mask corresponding to the invisible key point can be set to a second preset value (for example, 0); then, the three-dimensional axis angle of the key point in the third action gesture frame sequence information and the corresponding key point mask are superimposed to obtain the action gesture frame sequence sample information, wherein, when the key point is invisible, the three-dimensional axis angle corresponding to the key point can be set to the second preset value (for example, 0). Such a setting can prevent the action gesture information completion model from directly outputting the three-dimensional axis angle of the invisible key point, and set the three-dimensional axis angle of the invisible key point to 0, so that the action gesture information completion model can predict the three-dimensional axis angle of the invisible key point during training.

[0087] Taking each action posture frame including 24 key points and the action posture frame sequence including T time-sequential action posture frames as an example, during training, the relative position information of the key points of the input action posture frame sequence is a real number set of T*72 dimensions, that is, D∈R T×72 , where T is the number of temporally continuous action gesture frames, 72 represents the dimension of all three-dimensional axis angles in an action gesture frame (72 = 24*3), and the mask corresponding to the key point can be a 24-dimensional real number set, such as M∈R T×24 , where 24 corresponds to the number of key points, where 1 can be used to indicate that the key point is visible and 0 can be used to indicate that the key point is invisible. By superimposing the relative position information of the key points of the action posture frame sequence with the mask corresponding to the key points, a real number set of T*96 dimensions can be obtained, such as I∈R T×96 , which means that the action posture frame sequence sample information is a real number set of T*96 dimensions.

[0088] In the technical solution provided in the embodiment of the present application, on the one hand, it is necessary to consider using more time-sequentially continuous motion posture frames for prediction to improve the prediction accuracy of the relative position information of invisible key points. On the other hand, it is also necessary to consider the occupation of computing resources when the number of time-sequentially continuous motion posture frames is too large. Combining these two considerations, preferably, T can be set to 8.

[0089] Step S133, inputting the action posture frame sequence sample information into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information.

[0090] Please refer to Figure 3The generator may include an encoder and a decoder, wherein the encoder and the decoder are both composed of a multi-layer fully connected network, the encoder is connected to the decoder, the encoder performs dimensionality reduction processing on the data input to the generator, and the decoder performs dimensionality increase processing on the data output by the encoder. Step S133 can be implemented in the following way.

[0091] First, the action posture frame sequence sample information is input into the encoder for feature extraction.

[0092] Taking the above-mentioned 8*96-dimensional action posture frame sequence sample information as an example, the encoder can be a three-layer fully connected network, and the encoder extracts features from the input action posture frame sequence sample information by dimensionality reduction. Specifically, the dimension of the input data of the first layer of the fully connected network of the encoder is 8*96 dimensions, and the dimension of the output data of the first layer of the fully connected network of the encoder is 512 dimensions; the dimension of the input data of the second layer of the fully connected network of the encoder is 512 dimensions, and the dimension of the output data of the second layer of the fully connected network of the encoder is 512 dimensions; the dimension of the input data of the third layer of the fully connected network of the encoder is 512 dimensions, and the dimension of the output data of the third layer of the fully connected network of the encoder is 256 dimensions.

[0093] Next, the extracted features are input into the decoder for feature reconstruction to obtain reconstructed features.

[0094] Specifically, in the example of the above-mentioned 8*96-dimensional motion gesture frame sequence sample information, the decoder can also be a three-layer fully connected network, and the decoder reconstructs features by performing dimensionality-upgrading processing on the input data. Specifically, the input data of the first-layer fully connected network of the decoder is the output data of the third-layer fully connected network of the encoder (the dimension of the data is 256 dimensions), and the dimension of the output data of the first-layer fully connected network of the decoder is 512 dimensions; the dimension of the input data of the second-layer fully connected network of the decoder is 512 dimensions, and the dimension of the output data of the second-layer fully connected network of the decoder is 512 dimensions; the dimension of the input data of the third-layer fully connected network of the decoder is 512 dimensions, and the dimension of the output data of the third-layer fully connected network of the decoder is 8*96 dimensions. That is, the data output by the decoder has the same dimension as the data input by the encoder.

[0095] Finally, the relative position information of the visible key points in the third action posture frame sequence information is used to replace the relative position information of the corresponding visible key points in the reconstructed feature to obtain an updated reconstructed feature, and the updated reconstructed feature is used as the completed action posture frame sequence information.

[0096] It is understandable that the above example is only a possible architecture of a generator that can realize action posture frame sequence information completion provided by an embodiment of the present application. In other embodiments, the architecture of the generator can also be adjusted accordingly, for example, the number of fully connected network layers of the encoder and decoder, and the dimensions of the input and output data corresponding to each layer of the fully connected network can be adjusted accordingly. In some embodiments, the dimension of the output data of the encoder can be selected between 512 dimensions and 32 dimensions, that is, the dimension of the output data of the third layer of the encoder's fully connected network can be selected from 512 dimensions, and can also be selected from 128 dimensions, 64 dimensions or 32 dimensions. The input data of the first layer of the fully connected network of the decoder can also be selected from 512 dimensions to 32 dimensions, that is, the dimension of the input data of the first layer of the fully connected network of the decoder can be selected from 512 dimensions, and can also be selected from 128 dimensions, 64 dimensions or 32 dimensions.

[0097] Step S134 , judging whether the motion posture completion model is converged based on the third motion posture frame sequence information and the completed motion posture frame sequence information.

[0098] When it is determined that the action posture completion model has converged, the process proceeds to step S135 ; when it is determined that the action posture completion model has not converged, the process proceeds to step S136 .

[0099] In step S134, the first loss function value of the generator can be calculated based on the third motion posture frame sequence information and the completed motion posture frame sequence information, and whether the motion posture completion model converges can be determined based on the first loss function value of the generator.

[0100] Specifically, the calculation formula of the first loss function value Lg of the generator is:

[0101]

[0102] Wherein, T is the number of time series frames, t is the t-th frame in the time series frame, G is the number of key points of the target object, g represents the g-th key point, M(t,g) is the mask of the g-th key point in the t-th frame, M(t,g) is a first preset value (for example, 1) when the key point is visible, and M(t,g) is a second preset value (for example, 0) when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, It is the predicted relative position information of the key points in the completed action posture frame sequence information.

[0103] Furthermore, in a possible implementation, when the first loss function value Lg of the generator is less than a preset first loss function threshold, it is determined that the action posture completion model has converged; otherwise, it is determined that the action posture completion model has not converged.

[0104] Step S135, obtaining a trained action posture completion model.

[0105] Step S136, adjust the network parameters of the generator, and then return to step S131 to perform the next round of training process.

[0106] In this embodiment, the motion posture completion model obtained based on the above training process can predict the relative position relationship of invisible key points based on the relative position relationship of visible key points in multiple time-series continuous motion posture frames, so that the completed motion posture frame sequence information is basically consistent with the actual motion posture of the target object.

[0107] Furthermore, in order to make the completed action posture frame sequence information more consistent with the posture action of the target object, please refer to Figure 4 and Figure 5 , Figure 4 shows another step flow diagram of training the action posture completion model, Figure 5 Another model structure diagram of the action posture completion model is shown. In another possible implementation of the embodiment of the present application, the action posture completion model may include a generator and a discriminator. The encoder and the decoder are composed of a multi-layer fully connected network with the same number of layers. The encoder is connected to the decoder. The encoder performs dimensionality reduction processing on the data input to the generator, and the decoder performs dimensionality increase processing on the data output by the encoder. The discriminator is composed of a multi-layer fully connected network, and the discriminator is connected to the generator. Figure 4 and Figure 5 A detailed introduction to the steps of training a pose completion model.

[0108] Step S141, obtaining third motion posture frame sequence information and motion posture time sequence frame information of the target object.

[0109] The method of obtaining the third action posture frame sequence information of the target object is the same as step S131, which will not be repeated here.

[0110] In this step, the obtained motion gesture time sequence frame information may be real motion gesture sequence frame information obtained by sampling the motion gesture data collected by the motion capture device.

[0111] Step S142, obtaining action posture frame sequence sample information based on the invisible probability of the key point in the target object and the third action posture frame sequence information.

[0112] The method of obtaining the action posture frame sequence sample information is the same as step S132, which will not be repeated here.

[0113] Step S143, inputting the action posture frame sequence sample information into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information.

[0114] Among them, the process of the generator predicting the relative position information of the invisible key points in the action posture frame sequence sample information is the same as step S133 and will not be repeated here.

[0115] Step S144, inputting the completed motion gesture frame sequence information and the motion gesture time sequence frame information as input information into a discriminator for training, and obtaining a discriminant probability that the input information output by the discriminator is incomplete motion gesture frame sequence information.

[0116] In an embodiment of the present application, taking 8*96 dimensional action posture frame sequence information as an example, the discriminator can be a three-layer fully connected network. Specifically, the dimension of the input data of the first layer of the fully connected network of the discriminator is 8*96 dimensions, and the dimension of the output data of the first layer of the fully connected network of the discriminator is 512 dimensions; the dimension of the input data of the second layer of the discriminator is 512 dimensions, and the dimension of the output data of the second layer of the discriminator is 512 dimensions; the dimension of the input data of the third layer of the discriminator is 512 dimensions, and the dimension of the output data of the third layer of the discriminator is 1 dimension, wherein the 1-dimensional data is the discrimination probability that the input information is non-completed (true) action posture frame sequence information.

[0117] Specifically, through the adversarial network composed of the generator and the discriminator, the discriminator is more inclined to output the probability of 1 for the real action posture sequence frame information obtained by sampling, and is more inclined to output the probability of 0 for the completed action posture sequence frame information generated by the generator. Through the adversarial network, the completed action posture sequence frame information generated by the generator is eventually made close to the real action posture sequence frame information obtained by sampling, so that the completed action posture sequence frame information generated by the generator makes the discriminator output the probability of 1.

[0118] Step S145, based on the first loss function value of the generator, the discrimination probability of the discriminator for the action posture timing frame information, and the discrimination probability of the discriminator for the completed action posture frame sequence information, calculate the second loss function value of the generator and the loss function value of the discriminator.

[0119] Specifically, the second loss function value LG of the generator is calculated by the following calculation formula:

[0120] L G =L g +(D(G(input))-1) 2

[0121] Among them, the calculation formula of the generator's first loss function value Lg is:

[0122]

[0123] The calculation formula of the loss function value Ld of the discriminator is as follows:

[0124] L d =(1-D(gt)) 2 +(D(G(input))) 2

[0125] Wherein, T is the number of time series frames, t is the t-th frame in the time series frame, G is the number of key points of the target object, g represents the g-th key point, M(t,g) is the mask of the g-th key point in the t-th frame, M(t,g) is a first preset value (for example, 1) when the key point is visible, and M(t,g) is a second preset value (for example, 0) when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, is the predicted relative position information of the key points in the completed action posture frame sequence information, input is the action posture frame sequence sample information, G(input) is the completed action posture frame sequence information output by the generator, D(G(input)) is the discriminant probability of the completed action posture frame sequence information, gt is the action posture timing frame information, and D(gt) is the discriminant probability of the action posture timing frame information.

[0126] Step S146, determine whether the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition.

[0127] Specifically, when the second loss function value of the generator is less than the second loss function domain value and the loss function value of the discriminator is less than the third loss function domain value, it is determined that the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition; otherwise, it is determined that the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition. When it is determined that the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition, step S147 is entered; when it is determined that the second loss function value of the generator and the loss function value of the discriminator do not meet the convergence condition, step S148 is entered.

[0128] Step S147, obtaining a trained action posture completion model.

[0129] Step S148, adjust the network parameters in the generator and the discriminator, return to step S141, and perform the next round of training process.

[0130] In this embodiment, the motion posture completion model obtained through the above-mentioned training process can make the completed motion posture frame sequence information more consistent with the posture movement of the target object. For example, for some target objects (such as anchors) whose lower body is invisible but the upper body is visible, the predicted motion posture frame sequence information can include rich relative position relationships of the key points of the lower body.

[0131] Please refer to Figure 6 The action posture information completion method provided in the embodiment of the present application may further include step S13.

[0132] Step S13: driving the gesture action of the virtual image corresponding to the target object based on the second action gesture frame sequence information.

[0133] Based on the relative position relationship of visible key points in multiple action posture frames in time sequence in the action posture sequence information, the relative position relationship of invisible key points is predicted to obtain second action posture frame sequence information; and the posture movement of the virtual image corresponding to the target object is driven according to the second action posture frame sequence information, so that the virtual image can follow the posture movement of the target object and perform corresponding movements, so that in the transition stage when all key points of the target object are visible and some key points of the target object are invisible, the movement posture of the virtual object is smoother, thereby improving the live broadcast effect.

[0134] In summary, the motion posture information completion method provided in the embodiment of the present application predicts the relative position relationship of invisible key points based on the relative position relationship of visible key points in multiple motion posture frames in the time sequence in the motion posture sequence information, so as to ensure that the relative position relationship of invisible key points is predicted by the trained motion posture completion model based on the change of the relative position relationship of the key points in the time sequence, so as to obtain the completed second motion posture frame sequence information. The above-mentioned motion posture information completion method can be used as an independent module alone, and can be coupled with a previous model (such as a key point detection model) to achieve good motion posture information completion when some parts of the target object are invisible. In addition, the above-mentioned motion posture information completion method is simpler than the network structure of the existing motion posture information completion model (such as ResNet50 network and recurrent neural network), and the corresponding amount of calculation is much smaller (for example, about 10% of the amount of calculation of the existing motion posture information completion model). After introducing the adversarial network of the generator and the discriminator, the virtual image driven by the completed second action posture frame sequence information is more consistent with the posture changes of the target object, making the movement posture of the virtual object smoother in the transition stage where all the key points of the target object are visible and some of the key points of the target object are invisible, thereby improving the live broadcast effect.

[0135] For further information, please refer to Figure 7 , Figure 7A schematic diagram of a functional module of the motion posture information completion device 200 provided in an embodiment of the present application, in which the motion posture information completion device 200 can be divided into functional modules according to the method embodiment executed by the electronic device, that is, the following functional modules corresponding to the motion posture information completion device 200 can be used to execute the above-mentioned method embodiments. Among them, the motion posture information completion device 200 can include an acquisition module 210 and a completion module 220, and the functions of each functional module of the motion posture information completion device 200 are described in detail below.

[0136] The acquisition module 210 is used to acquire first motion posture frame sequence information of the target object.

[0137] In the embodiment of the present application, the target object can be a person (eg, a webcaster) or other animals (eg, a panda).

[0138] Wherein, the first motion gesture frame sequence information can be composed of information of multiple motion gesture frames that are continuous in time sequence, and the information of each motion gesture frame includes visibility information of each key point in the target object and relative position information of visible key points among each key point in the motion gesture frame. Wherein, the visibility information can include indication information of whether the key point is visible, for example, including first indication information that the key point is visible and second indication information that the key point is invisible. Specifically, the visibility information can be represented by a digital mark, for example, for a visible key point, the number 1 can be used to represent the visibility information of the key point, and for an invisible key point, the number 0 can be used to represent the visibility information of the key point. Wherein, the motion gesture frame can be obtained by a motion capture device, and the motion capture device can include an optical motion capture device and a thermal imaging motion capture device, etc. The relative position information of the key point refers to information used to confirm the relative position of the key point with other key points, wherein other key points can be adjacent key points of the key point, or a reference key point.

[0139] The acquisition module 210 may be used to execute the above step S11. For the detailed implementation of the acquisition module 210, reference may be made to the above detailed description of step S11.

[0140] The completion module 220 is used to input the first motion posture frame sequence information into the trained motion posture completion model for prediction, and complete the relative position information of the invisible key points in the first motion posture frame sequence information to obtain the second motion posture frame sequence information.

[0141] Specifically, the trained action posture completion model can predict the relative position information of the invisible key points according to the relative position information of the visible key points in each action posture frame that is consecutive in time sequence in the first action posture frame sequence information.

[0142] The completion module 220 may be used to execute the above step S12. For the detailed implementation of the completion module 220, reference may be made to the above detailed description of step S12.

[0143] Please refer to Figure 8 Another functional module schematic diagram of the motion posture information completion device 200 provided in an embodiment of the present application.

[0144] In the embodiment of the present application, the motion posture information completion device 200 may further include a training module 230 .

[0145] In one implementation of the embodiment of the present application, the action posture completion model includes a generator, and the training module 230 can be used to:

[0146] Acquire third action posture frame sequence information of the target object;

[0147] Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information;

[0148] The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information;

[0149] Based on the action posture frame sequence sample information and the completed action posture frame sequence information, it is judged whether the action posture completion model has converged. If it has not converged, the network parameters of the generator are adjusted and the step of obtaining the action posture frame sequence sample information is returned to until the action posture completion model converges to obtain a trained action posture completion model.

[0150] In this implementation, the training module 230 may be used to execute the above-mentioned steps S131 to S136. The detailed implementation of the training module 230 may refer to the above-mentioned detailed description of steps S131 to S136.

[0151] In another implementation of the embodiment of the present application, the action posture completion model includes a generator and a discriminator, and the training module 230 can be used to:

[0152] Acquire third motion posture frame sequence information and motion posture timing frame information of the target object;

[0153] Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information;

[0154] The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information;

[0155] Inputting the completed motion gesture frame sequence information and the motion gesture time sequence frame information as input information into the discriminator for training, and obtaining a discriminant probability that the input information output by the discriminator is non-completed motion gesture frame sequence information;

[0156] Calculate the second loss function value of the generator and the loss function value of the discriminator based on the first loss function value of the generator, the discrimination probability of the discriminator for the action posture time sequence frame information, and the discrimination probability of the discriminator for the completed action posture frame sequence information;

[0157] When the second loss function value of the generator and the loss function value of the discriminator do not meet the convergence condition, adjust the network parameters in the generator and the discriminator, and return to the step of obtaining the third action posture frame sequence information and action posture timing frame information of the target object until the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition.

[0158] In this implementation, the training module 230 may be used to execute the above-mentioned steps S141 to S148. The detailed implementation of the training module 230 may refer to the above-mentioned detailed description of steps S141 to S148.

[0159] Please refer again Figure 8 The motion posture information completion device 200 may further include a driving module 240 .

[0160] The driving module 240 is used to drive the posture action of the virtual image corresponding to the target object based on the second action posture frame sequence information.

[0161] It should be noted that it should be understood that the division of the various modules in the above device or system is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software (for example, open source software) that can be called by the processor; they can also be implemented in the form of hardware; some modules can also be implemented in the form of software called by the processor, and some modules can be implemented in the form of hardware. As an example, the training module 230 can be implemented by running a separate processor, and can be stored in the memory of the above device or system in the form of program code. The functions of the above training module 230 are called and executed by a processor of the above device or system. The implementation of other modules is similar to it, and it will not be repeated here. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processor described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step or each module in the above technical solution can be completed by an integrated logic circuit in the processor or by executing a software program.

[0162] Please refer to Fig. 9 , Fig. 9 FIG. 1 is a schematic diagram showing the hardware structure of an electronic device 10 for implementing the above-mentioned action posture information completion method provided by an embodiment of the present disclosure. Fig. 9 As shown, the electronic device 10 may include a processor 11 , a computer-readable storage medium 12 , and a bus 13 .

[0163] In a specific implementation process, the processor 11 executes the computer-executable instructions (eg, Figure 8 The motion posture information completion device 200 shown in the figure includes an acquisition module 210, a smoothing module 220, a determination module 230 and a synthesis module 240), so that the processor 11 can execute the motion posture information completion method of the above method embodiment, wherein the processor 11 and the computer-readable storage medium 12 can be connected through a bus 13.

[0164] The specific implementation process of the processor 11 can refer to the various method embodiments executed by the above-mentioned electronic device 10. The implementation principles and technical effects are similar and will not be repeated in this embodiment.

[0165] The computer readable storage medium 12 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 111 is used to store programs or data.

[0166] The bus 13 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0167] Above Fig. 9 This is only a possible structural diagram of the electronic device 10 provided in an embodiment of the present application. In other embodiments, the electronic device 10 may also include more components, such as a motion capture device, and the electronic device 10 may obtain motion posture frames through the motion capture device.

[0168] In addition, an embodiment of the present application further provides a readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the motion posture information completion method as described above is implemented.

[0169] In summary, the motion posture information completion method, electronic device and computer-readable storage medium provided by the embodiments of the present application predict the relative position relationship of invisible key points based on the relative position relationship of visible key points in multiple motion posture frames in the time sequence in the motion posture sequence information, so as to ensure that the relative position relationship of invisible key points is predicted by the trained motion posture completion model based on the change of the relative position relationship of the key points in the time sequence, so as to obtain the completed second motion posture frame sequence information. The above-mentioned motion posture information completion method can be used as an independent module alone, and can be coupled with a previous model (such as a key point detection model) to achieve good motion posture information completion when some parts of the target object are invisible. In addition, the above-mentioned motion posture information completion method is simpler than the existing motion posture information completion model network structure, and the corresponding calculation amount is much smaller (for example, about 10% of the calculation amount of the existing motion posture information completion model). After introducing the adversarial network of the generator and the discriminator, the virtual image driven by the completed second action posture frame sequence information is more consistent with the posture changes of the target object, making the movement posture of the virtual object smoother in the transition stage where all the key points of the target object are visible and some of the key points of the target object are invisible, thereby improving the live broadcast effect.

[0170] In addition, unless explicitly stated in the claims, the order of processing elements and sequences, the use of alphanumeric characters, or the use of other names in this specification are not intended to limit the order of the processes and methods of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0171] Similarly, it should be noted that in order to simplify the description disclosed in this specification and thus help understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the subject matter of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0172] The embodiments described above are only some embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application generally described and shown in the drawings can be arranged and designed in various configurations. Therefore, the detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of protection of the present application, but merely represents the selected embodiments of the present application. Based on this, the scope of protection of the present application shall be based on the scope of protection of the claims. In addition, based on the embodiments of the present application, all other embodiments that can be obtained by those skilled in the art without making creative work should belong to the scope of protection of the present application.

Claims

1. A method for completing action posture information, characterized in that: The method comprises: Acquire first motion gesture frame sequence information of the target object, wherein the first motion gesture frame sequence information includes visibility information of each key point in the target object in each motion gesture frame and relative position information of visible key points in each key point in each motion gesture frame, and there is at least one invisible key point in each key point, wherein the relative position information of the key point refers to information used to confirm the relative position of the key point with other key points, and the relative position information of the key point is a three-dimensional axis angle; The first motion posture frame sequence information is input into the trained motion posture completion model for prediction, and the relative position information of the invisible key points in the first motion posture frame sequence information is completed to obtain the second motion posture frame sequence information, wherein the trained motion posture completion model predicts the relative position information of the invisible key points based on the relative position information of the visible key points in each of the motion posture frames in time sequence.

2. The method for completing the action posture information according to claim 1, characterized in that: The action posture completion model includes a generator, which includes an encoder and a decoder. The encoder and the decoder are composed of a multi-layer fully connected network with the same number of layers. The encoder is connected to the decoder. The encoder performs dimensionality reduction processing on the data input into the generator, and the decoder performs dimensionality increase processing on the data output by the encoder.

3. The method for completing the action posture information according to claim 2, characterized in that: The dimension of the output data of the last fully connected network layer of the encoder includes one dimension from 512 dimensions to 32 dimensions.

4. The method for completing the action posture information according to claim 2, wherein: The method further comprises the step of training the action posture completion model, the step comprising: Acquire third action posture frame sequence information of the target object; Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information; The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information; Based on the third motion posture frame sequence information and the completed motion posture frame sequence information, determine whether the motion posture completion model has converged. If it has not converged, adjust the network parameters of the generator and return to the step of obtaining the third motion posture frame sequence information of the target object until the motion posture completion model converges to obtain a trained motion posture completion model.

5. The method for completing the action posture information according to claim 4, characterized in that: The step of obtaining the action posture frame sequence sample information based on the invisible probability of the key point in the target object and the third action posture frame sequence information comprises: Determining a key point mask corresponding to a key point in each action posture frame in the third action posture frame sequence information based on the invisible probability of different key points in the target object, wherein the key point mask corresponding to the visible key point is set to a first preset value, and the key point mask corresponding to the invisible key point is set to a second preset value; The three-dimensional axis angle of the key point in the third action posture frame sequence information and the corresponding key point mask are superimposed to obtain the action posture frame sequence sample information, wherein when the key point is not visible, the three-dimensional axis angle corresponding to the key point is set to the second preset value.

6. The method for completing the action posture information according to claim 5, characterized in that: The step of predicting the relative position information of the invisible key points in the action posture frame sequence sample information by the generator to obtain the completed action posture frame sequence information includes: Inputting the action posture frame sequence sample information into the encoder for feature extraction; Inputting the extracted features into the decoder to perform feature reconstruction to obtain reconstructed features; The relative position information of the visible key points in the third action posture frame sequence information is used to replace the relative position information of the corresponding visible key points in the reconstructed feature to obtain an updated reconstructed feature, and the updated reconstructed feature is used as the completed action posture frame sequence information.

7. The method for completing the action posture information according to claim 6, characterized in that: The step of judging whether the action posture completion model has converged based on the third action posture frame sequence information and the completed action posture frame sequence information comprises: Calculating a first loss function value of a generator based on the third action posture frame sequence information and the completed action posture frame sequence information, and judging whether the action posture completion model converges based on the first loss function value of the generator; The calculation formula of the first loss function value Lg of the generator is: Wherein, T is the number of time series frames, t is the t-th frame in the time series frame, G is the number of key points of the target object, g represents the g-th key point, M(t,g) is the mask of the g-th key point in the t-th frame, M(t,g) is the first preset value when the key point is visible, and M(t,g) is the second preset value when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, It is the predicted relative position information of the key points in the completed action posture frame sequence information.

8. The method for completing the action posture information according to claim 1, characterized in that: The action posture completion model includes an adversarial network composed of a generator and a discriminator, the generator includes an encoder and a decoder, the encoder and the decoder are composed of multi-layer fully connected networks with the same number of layers, the encoder is connected to the decoder, the encoder performs dimensionality reduction processing on the data input to the generator, and the decoder performs dimensionality increase processing on the data output by the encoder, the discriminator is composed of a multi-layer fully connected network, and the discriminator is connected to the generator.

9. The method for completing the action posture information according to claim 8, characterized in that: The method further comprises the step of training the action posture completion model, the step comprising: Acquire third motion posture frame sequence information and motion posture timing frame information of the target object; Based on the invisible probability of the key point in the target object and the third action posture frame sequence information, obtaining the action posture frame sequence sample information; The action posture frame sequence sample information is input into the generator for training, and the generator predicts the relative position information of the invisible key points in the action posture frame sequence sample information to obtain the completed action posture frame sequence information; Inputting the completed motion gesture frame sequence information and the motion gesture time sequence frame information as input information into the discriminator for training, and obtaining a discriminant probability that the input information output by the discriminator is non-completed motion gesture frame sequence information; Calculate the second loss function value of the generator and the loss function value of the discriminator based on the first loss function value of the generator, the discrimination probability of the discriminator for the action posture time sequence frame information, and the discrimination probability of the discriminator for the completed action posture frame sequence information; When the second loss function value of the generator and the loss function value of the discriminator do not meet the convergence condition, adjust the network parameters in the generator and the discriminator, and return to the step of obtaining the third action posture frame sequence information and action posture timing frame information of the target object until the second loss function value of the generator and the loss function value of the discriminator meet the convergence condition.

10. The method for completing the action posture information according to claim 9, characterized in that: The first loss function value Lg of the generator is calculated by the following calculation formula: The second loss function value LG of the generator is calculated by the following calculation formula: L G =L g +(D(G(input))-1) 2 The calculation formula of the loss function value Ld of the discriminator is as follows: L d =(1-D(gt)) 2 +(D(G(input))) 2 Wherein, T is the number of time series frames, t is the t-th frame in the time series frame, G is the number of key points of the target object, g represents the g-th key point, M(t,g) is the mask of the g-th key point in the t-th frame, M(t,g) is the first preset value when the key point is visible, and M(t,g) is the second preset value when the key point is invisible. is the real relative position information of the key points in the third action posture frame sequence information, is the predicted relative position information of the key points in the completed action posture frame sequence information, input is the action posture frame sequence sample information, G(input) is the completed action posture frame sequence information output by the generator, D(G(input) is the discriminant probability of the completed action posture frame sequence information, gt is the action posture time sequence frame information, and D(gt) is the discriminant probability of the action posture time sequence frame information.

11. The method for completing the action posture information according to any one of claims 1 to 10, characterized in that: After the step of inputting the first action posture frame sequence information into a trained action posture completion model for training, completing the relative position information of invisible key points in the first action posture frame sequence information, and obtaining the second action posture frame sequence information, the method further includes: The gesture action of the virtual image corresponding to the target object is driven based on the second action gesture frame sequence information.

12. An electronic device, characterized in that: The electronic device includes a processor and a computer-readable storage medium, the processor and the computer-readable storage medium are connected via a bus system, the computer-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the computer-readable storage medium to implement the motion posture information completion method described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions which, when executed, enable the electronic device to execute the motion posture information completion method described in any one of claims 1-11.

Images