Action processing method and apparatus

Abstract

The embodiment of the specification provides a motion processing method and device, wherein the motion processing method comprises the following steps: obtaining a reference motion frame sequence, and generating an initial motion frame sequence; filling the initial motion frame sequence into the reference motion frame sequence according to a preset filling mode to obtain a to-be-processed motion frame sequence; performing denoising processing and motion detection processing on the motion frame based on the to-be-processed motion frame sequence and motion feature parameters by means of a motion processing model to obtain a candidate motion frame sequence and motion interaction data; and performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain a target motion frame sequence.

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to an action processing method and apparatus. Background Technology

[0002] An action frame is an image frame that describes the action or posture of an object. With the rapid development of internet technology, more and more fields are beginning to conduct in-depth research on action frames, such as the film and game industries, which need to use action frames to describe the actions of objects and to create animations. In this process, users' demand for cultural entertainment is also growing, and film, animation, and games account for a large part of cultural entertainment. As a result, the producers of action frames are also under increasing pressure in their production. Summary of the Invention

[0003] This specification provides one or more embodiments of an action processing method, comprising: acquiring a baseline action frame sequence and generating an initial action frame sequence; filling the baseline action frame sequence with the initial action frame sequence according to a preset filling method to obtain an action frame sequence to be processed; inputting the action frame sequence to be processed and action feature parameters into an action processing model for denoising and action detection processing of the action frames to obtain a candidate action frame sequence and action interaction data; and performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain a target action frame sequence.

[0004] This specification provides one or more embodiments of a model training method, comprising: inputting action frame sequence samples into a model to be trained for action frame denoising and action detection processing to obtain a predicted action frame sequence and predicted action interaction data. The action frame sequence samples are obtained after adding noise to target action frames in a preset action frame sequence. A first loss is calculated based on the target action frames and the target predicted action frames in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frames and the predicted action interaction data. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain an action processing model.

[0005] This specification provides one or more embodiments of an action processing apparatus, comprising: an action frame sequence acquisition module configured to acquire a reference action frame sequence and generate an initial action frame sequence; an action frame filling module configured to fill the initial action frame sequence into the reference action frame sequence according to a preset filling method to obtain an action frame sequence to be processed; a denoising processing module configured to input the action frame sequence to be processed and action feature parameters into an action processing model for denoising and action detection processing of the action frames to obtain a candidate action frame sequence and action interaction data; and an action correction module configured to perform action correction processing on the candidate action frame sequence based on the action interaction data to obtain a target action frame sequence.

[0006] This specification provides one or more embodiments of a model training apparatus, comprising: a sample input module configured to input action frame sequence samples into a model to be trained for action frame denoising and action detection processing to obtain a predicted action frame sequence and predicted action interaction data. The action frame sequence samples are obtained after adding noise to target action frames in a preset action frame sequence. A loss calculation module configured to calculate a first loss based on the target action frames and the target predicted action frames in the predicted action frame sequence, and to calculate a second loss based on the target predicted action frames and the predicted action interaction data. A parameter adjustment module configured to adjust the parameters of the model to be trained based on the first loss and the second loss to obtain an action processing model.

[0007] This specification provides one or more embodiments of an action processing device, including: a processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: acquire a baseline action frame sequence and generate an initial action frame sequence; fill the baseline action frame sequence with the initial action frame sequence according to a preset filling method to obtain a sequence of action frames to be processed; input the sequence of action frames to be processed and action feature parameters into an action processing model for denoising and action detection processing of the action frames to obtain a candidate action frame sequence and action interaction data; and perform action correction processing on the candidate action frame sequence based on the action interaction data to obtain a target action frame sequence.

[0008] This specification provides one or more embodiments of a model training device, comprising: inputting action frame sequence samples into a model to be trained for action frame denoising and action detection processing to obtain a predicted action frame sequence and predicted action interaction data. The action frame sequence samples are obtained after adding noise to target action frames in a preset action frame sequence. A first loss is calculated based on the target action frames and the target predicted action frames in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frames and the predicted action interaction data. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain an action processing model.

[0009] This specification provides one or more embodiments of a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring a baseline action frame sequence and generating an initialization action frame sequence; filling the baseline action frame sequence with the initialization action frame sequence according to a preset filling method to obtain a sequence of action frames to be processed; inputting the sequence of action frames to be processed and action feature parameters into an action processing model for denoising and action detection processing to obtain a candidate action frame sequence and action interaction data; and performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain a target action frame sequence.

[0010] One or more embodiments of this specification provide another storage medium for storing computer-executable instructions, which, when executed by a processor, implement the following process: inputting action frame sequence samples into a model to be trained for action frame denoising and action detection processing to obtain a predicted action frame sequence and predicted action interaction data. The action frame sequence samples are obtained after adding noise to target action frames in a preset action frame sequence. A first loss is calculated based on the target action frames and the target predicted action frames in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frames and the predicted action interaction data. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain an action processing model. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in one or more embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1A schematic diagram of the implementation environment for one or more embodiments of the action processing method provided in this specification;

[0013] Figure 2 A flowchart of an action processing method provided for one or more embodiments of this specification;

[0014] Figure 3 A schematic diagram of a target action frame sequence provided for one or more embodiments of this specification;

[0015] Figure 4 A flowchart illustrating a motion processing method for motion frame scenes provided in one or more embodiments of this specification;

[0016] Figure 5 A flowchart illustrating a model training method provided in one or more embodiments of this specification;

[0017] Figure 6 A flowchart illustrating a model training method for action frame scenes provided in one or more embodiments of this specification;

[0018] Figure 7 A schematic diagram of an embodiment of a motion processing device provided in one or more embodiments of this specification;

[0019] Figure 8 A schematic diagram of an embodiment of a model training device provided in one or more embodiments of this specification;

[0020] Figure 9 This specification provides a schematic diagram of the structure of a motion processing device according to one or more embodiments.

[0021] Figure 10 This is a schematic diagram of the structure of a model training device provided in one or more embodiments of this specification. Detailed Implementation

[0022] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0023] Reference Figure 1 This specification provides a schematic diagram of the implementation environment for one or more embodiments of the action processing method.

[0024] The motion processing method provided in one or more embodiments of this specification is applicable to the implementation environment of motion frame processing, which includes at least a server 101 for performing motion frame processing.

[0025] In addition, the implementation environment may also include a terminal device 102, which can be configured to interact with a client that interacts with the server 101. The client may be an application, a subroutine within an application, a service module within an application, or a web application.

[0026] Server 101 can be a single server, multiple servers, a server cluster consisting of several servers, or a cloud server of a cloud computing platform, used to generate a target action frame sequence based on a baseline action frame sequence and an initial action frame sequence. An action processing model can be deployed on server 101, used for denoising and action detection of the action frames.

[0027] Terminal device 102 can be a mobile phone, personal computer, tablet computer, e-book reader, device for information interaction based on VR (Virtual Reality) technology, vehicle terminal, IoT device, wearable smart device, laptop computer and desktop computer, etc. Terminal device 102 can be used to submit a reference action frame sequence to server 101.

[0028] In this implementation environment, server 101 can acquire a baseline action frame sequence and generate an initial action frame sequence. The initial action frame sequence is then filled into the baseline action frame sequence according to a preset filling method to obtain a sequence of action frames to be processed. Using an action processing model, denoising and action detection are performed on the action frames based on the sequence of action frames to be processed and action feature parameters to obtain candidate action frame sequences and action interaction data. Based on the action interaction data, action correction processing is performed on the candidate action frame sequences to obtain the target action frame sequence. This action correction processing avoids discontinuities between action frames in the target action frame sequence, improving the smoothness and naturalness of each action frame. Simultaneously, action feature parameters enable diversity and flexibility in the target action frame sequence, making the action frames in the target action frame sequence closer to real action frames, thus improving the effectiveness and realism of the target action frame sequence.

[0029] One or more embodiments of an action processing method provided in this specification are as follows:

[0030] Reference Figure 2 The action processing method provided in this embodiment specifically includes steps S202 to S208.

[0031] Step S202: Obtain the baseline action frame sequence and generate the initialization action frame sequence.

[0032] The reference action frame sequence described in this embodiment refers to a sequence composed of one or more reference action frames. Optionally, the reference action frame sequence includes a start action frame and / or a stop action frame. Furthermore, the reference action frame sequence may also include a first action frame and / or a second action frame. In this embodiment, an action frame refers to an image frame used to characterize an object's action or posture, such as dancing, running, or playing ball. Additionally, the object's action or posture can be other types of actions or postures. The object here can be a real-world person, a virtual person, or other creatures capable of producing actions or postures, such as animals or marine life. Figure 3 The diagram shown is a schematic representation of the target action frame sequence. Figure 3 (1) It can be the starting action frame, Figure 3 (4) It can be a termination action frame.

[0033] It should be noted that the starting action frame here refers to the image frame that serves as the starting action or starting posture. Similarly, the ending action frame refers to the image frame that serves as the ending action or ending posture. The starting action frame can be one or more frames, and the ending action frame can also be one or more frames. In this embodiment, the action frame can be represented by a feature sequence of m joint parts. Each joint part is specifically represented by n degrees of freedom or rotation. The rotation can be the deflection angle or rotation angle of each joint part relative to the due south, due north, due west, or due east direction. In addition, the rotation can also be the deflection angle or rotation angle of each joint part relative to the target direction. For the key parts in each action frame, b can be used to represent the action interaction state.

[0034] The initialization action frame sequence refers to a sequence composed of one or more initialization action frames; the initialization action frames may be randomly generated noisy action frames. In an optional implementation of this embodiment, the following operations are performed during the process of obtaining the reference action frame sequence and generating the initialization action frame sequence:

[0035] Get the start and end action frames;

[0036] A preset number of random numbers are sampled from a Gaussian distribution, and the initialization action frame sequence is generated based on the sampled random numbers.

[0037] Specifically, the starting action frame and the ending action frame are obtained, and a baseline action frame sequence is constructed based on the starting action frame and the ending action frame. A preset number of random numbers are sampled from a Gaussian distribution, and an initialization action frame is constructed based on the sampled random numbers. An initialization action frame sequence is constructed based on the initialization action frame.

[0038] For example, to obtain the start and end action frames, 24*3 random numbers are sampled in a Gaussian distribution of 0-1. An initial action frame can be constructed based on the sampled 24*3 random numbers, and an initial action frame sequence can be constructed based on the constructed initial action frames. It should be noted that since each action frame can be represented by 3 rotation degrees of 24 joint parts, an initial action frame can be constructed by sampling 24*3 random numbers here.

[0039] In addition, during the process of acquiring the baseline action frame sequence and generating the initial action frame sequence, the start action frame or the end action frame can also be acquired; a preset number of random numbers are sampled from the Gaussian distribution, and the initial action frame sequence is generated based on the sampled random numbers.

[0040] In practical applications, there is often a need to generate transition frames. For example, given the start and end action frames of a dance, there is a need to generate a continuous and natural sequence of dance action frames based on the start and end action frames. To address this, a baseline action frame sequence can be obtained, and an initial action frame sequence can be generated.

[0041] In specific implementation, in addition to the methods of obtaining the baseline action frame sequence and generating the initial action frame sequence based on sampled random numbers, after obtaining the baseline action frame sequence, a preset number of random numbers can be sampled in a Gaussian distribution based on the baseline action frame sequence, and the initial action frames can be reconstructed based on the sampled random numbers. The initial action frame sequence can then be constructed based on the reconstructed initial action frames.

[0042] Step S204: Fill the initial action frame sequence into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed.

[0043] In the above steps of obtaining the baseline action frame sequence and generating the initialization action frame sequence, the initialization action frame sequence can be filled into the baseline action frame sequence to obtain the action frame sequence to be processed.

[0044] The preset filling method described in this embodiment refers to a pre-set filling method for the base action frame sequence, such as filling the initialization action frame sequence to the middle filling position of the base action frame sequence.

[0045] In specific implementation, in order to obtain a continuous and natural target action frame sequence, an action frame sequence with the same number of frames as the target action frame sequence can be obtained first to form the basis for the output target action frame sequence. In an optional implementation provided in this embodiment, during the process of filling the initial action frame sequence into the base action frame sequence according to a preset filling method to obtain the action frame sequence to be processed, the following operations are performed:

[0046] The initial action frame sequence is filled into the middle padding position between the start action frame and the end action frame to obtain the action frame sequence to be processed.

[0047] Optionally, the intermediate padding bit refers to the middle position between the start action frame and the end action frame.

[0048] In addition to the aforementioned implementation where the padding is positioned between the start and end action frames, the aforementioned reference action frame sequence also includes a first action frame and / or a second action frame. In this case, the padding for filling the initialization action frame sequence can be positioned before the first action frame, after the second action frame, or between the first and second action frames. During the process of filling the initialization action frame sequence into the reference action frame sequence according to a preset padding method to obtain the action frame sequence to be processed, the following operations can be performed:

[0049] Determine the padding bits for each initialization action frame in the initialization action frame sequence within the base action frame sequence;

[0050] Each initialization action frame is filled into the padding bits in the reference action frame sequence to obtain the action frame sequence to be processed.

[0051] For example, in the initialization action frame sequence, the padding bit of the first initialization action frame in the first action frame and the second action frame is determined to be the first padding bit before the first action frame; the padding bit of the second initialization action frame in the first action frame and the second action frame is determined to be the second padding bit between the first action frame and the second action frame; and the padding bit of the third initialization action frame in the first action frame and the second action frame is determined to be the third padding bit after the second action frame. The first initialization action frame is filled to the first padding bit, the second initialization action frame is filled to the second padding bit, and the third initialization action frame is filled to the third padding bit to obtain the action frame sequence to be processed.

[0052] Step S206: Input the action frame sequence to be processed and the action feature parameters into the action processing model to perform denoising processing and action detection processing of the action frames, and obtain candidate action frame sequences and action interaction data.

[0053] In the above steps, the initial action frame sequence is filled into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed. In this step, in order to achieve the continuity and effectiveness of the target action frame sequence, the action processing model can perform denoising and action detection processing on the action frame based on the action frame sequence to be processed and action feature parameters to obtain the candidate action frame sequence and action interaction data.

[0054] The motion processing model described in this embodiment is used for denoising and motion detection of motion frames. It corrects the candidate motion frame sequence based on the obtained candidate motion frame sequence and motion interaction data to obtain the target motion frame sequence. In other words, the motion processing model generates a complete and continuous sequence of candidate motion frames based on a baseline motion frame sequence. For example, it generates a transition frame between the start and end motion frames. For instance, the motion processing model can be a diffusion model, such as a diffusion model with a DDPM (Denoising Diffusion Probabilistic Models) structure. DDPM is a generative model based on Markov noise diffusion and belongs to the diffusion model category. That is, the motion processing model described in this embodiment can adopt a diffusion model with a DDPM structure. This diffusion model is used for denoising and motion detection of motion frames to achieve the motion processing function. Furthermore, when the motion processing model is a diffusion model, other types of model structures can also be used besides the DDPM model structure.

[0055] The action feature parameters mentioned in this embodiment refer to parameters that characterize the action features of an object. Optionally, the action feature parameters include action attribute parameters and / or action type parameters. The action attribute parameters include parameters that characterize action attributes, and the action type parameters include parameters that characterize action types. For example, the action attribute parameters are dancing, playing basketball, or running, and the action type parameters are cheerful or strenuous actions. In addition, the action feature parameters, action attribute parameters, and action type parameters can also be other types of parameters. For example, the action feature parameters include the attribute parameters of the object to which the action belongs, that is, the attribute parameters of the object that produces the action. For example, if the object that produces the action is a real person, the attribute parameters of the object to which the action belongs can be the object's identity attribute (parameters for whether it is male or female) or the object's growth time.

[0056] The motion interaction data includes the motion interaction state between the key part and the target position, and the key part and / or the target position; optionally, the motion interaction state includes a contact state and / or a separation state. When the motion interaction state is in the contact state, it means that the key part and the target position are in contact. When the motion interaction state is in the separation state, it means that the key part and the target position are separated.

[0057] The key parts include key parts of the object; the key parts can be the foot parts, specifically the left toe, left heel, right toe and / or right heel; the target position refers to the target position in the action frame, such as the ground position in the action frame.

[0058] In practical implementation, since each initialization action frame in the initialization action frame sequence contains noise, in order to obtain an effective target action frame sequence, the initialization action frame sequence in the action frame sequence to be processed can be denoised. In an optional implementation provided in this embodiment, the denoising of the action frames is implemented in the following manner:

[0059] Based on the action feature parameters and the reference action frame sequence in the action frame sequence to be processed, noise prediction processing is performed on the initial action frame sequence in the action frame sequence to be processed.

[0060] The initial action frame sequence is subjected to noise reduction processing based on the noise prediction results to obtain the candidate action frame sequence containing the noise-reduced action frame sequence.

[0061] The noise-reduced action frame sequence here corresponds to the initial action frame sequence; the candidate action frame sequence corresponds to the action frame sequence to be processed.

[0062] Furthermore, in order to further reduce the noise content of the noise-reduced action frame sequence and improve the effectiveness of the candidate action frame sequence, the initial action frame sequence can be denoised multiple times. In this case, the denoising of the action frames can be achieved in the following way:

[0063] Based on the action feature parameters and the baseline action frame sequence in the action frame sequence to be processed, noise prediction processing is performed on the initial action frame sequence in the action frame sequence to be processed.

[0064] The initial action frame sequence is noise-removed based on the noise prediction results, and the specific action frame sequence after noise removal is denoised based on the action feature parameters and the baseline action frame sequence.

[0065] Check if the number of denoising attempts exceeds the preset number of denoising attempts; if yes, obtain a candidate action frame sequence containing the noise-reduced action frame sequence; if no, return to perform denoising processing on the noise-reduced specific action frame sequence based on action feature parameters and the baseline action frame sequence.

[0066] After denoising the action frame sequence, to eliminate slippage in the candidate action frame sequence obtained after denoising and improve the smoothness and flexibility of the target action frame sequence, action detection processing can be performed on the candidate action frame sequence. This involves determining the position indicators of key parts in the candidate action frame sequence and detecting whether the position indicators of the key parts meet preset indicator conditions. If so, the action interaction state between the key parts and the target position in the candidate action frame is determined to be a contact state, and action interaction data is constructed based on the contact state, key parts, and target position, or the contact state is used as the action interaction data. If not, the action interaction state between the key parts and the target position in the candidate action frame is determined to be a separation state, and action interaction data is constructed based on the separation state, key parts, and target position, or the separation state is used as the action interaction data. Specifically, in an optional implementation method provided in this embodiment, action detection processing is performed in the following manner:

[0067] Detect whether the positional deviation of the key parts of the candidate action frame and the adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or, detect whether the distance between the key parts of the candidate action frame and the target position is less than a preset distance threshold;

[0068] If the position deviation is less than the preset deviation threshold and / or the distance is less than the preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a contact state, and the action interaction data is constructed based on the contact state, the key part and the target position.

[0069] If the position deviation is greater than or equal to the preset deviation threshold and / or the distance is greater than or equal to the preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a separated state, and action interaction data is constructed based on the separated state, the key part and the target position.

[0070] Optionally, the key parts include the heel and / or toe, the heel including the left heel and / or right heel, and the toe including the left toe and / or right toe. The positional deviation includes the difference between the position of the key parts in the candidate action frame and their position in adjacent action frames. The target position can be the ground position. In this embodiment, the candidate action frames in the candidate action frame sequence can be any action frame in the candidate action frame sequence.

[0071] It should be noted that the implementation process of motion detection here can also be the specific implementation process of detecting whether the position indicators of key parts meet the preset indicator conditions. If so, the motion interaction state between the key parts and the target position in the candidate motion frame is determined to be a contact state, and the motion interaction data is constructed based on the contact state, key parts and target position.

[0072] Furthermore, during the motion detection processing of motion frames, motion detection processing can also be performed only on the noise-reduced motion frame sequence in the candidate motion frame sequence. Specifically, the following operations can be performed: determine the position indicators of key parts in the noise-reduced motion frame sequence of the candidate motion frame sequence, and detect whether the position coordinates of the key parts meet the preset indicator conditions. If so, determine that the motion interaction state between the key parts and the target position in the noise-reduced motion frame sequence is a contact state, and construct motion interaction data based on the contact state, the key parts, and the target position, or use the contact state as the motion interaction data; if not, determine that the motion interaction state between the key parts and the target position in the noise-reduced motion frame sequence is a separation state, and construct motion interaction data based on the separation state, the key parts, and the target position, or use the separation state as the motion interaction data.

[0073] It should be noted that the process of detecting whether the position coordinates of the key parts meet the preset index conditions can be achieved by performing the following methods: detecting whether the position deviation of the key parts between the candidate action frame and the adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or detecting whether the distance between the key parts in the candidate action frame and the target position is less than a preset distance threshold.

[0074] In specific implementation, while inputting the sequence of action frames to be processed and action feature parameters into the action processing model for denoising and action detection to obtain candidate action frame sequences and action interaction data, the sequence of action frames to be processed and action feature parameters can also be input into the action processing model for denoising to obtain a denoised specific action frame sequence. The denoised specific action frame sequence and action feature parameters are then input into the action processing model for further denoising, and the number of denoising attempts is checked to see if it exceeds the preset number of denoising attempts. If so, a candidate action frame sequence is output, and the candidate action frame sequence is input into the action processing model for action detection to obtain action interaction data. If not, the process returns to performing the denoised specific action frame sequence and action feature parameters input into the action processing model for further denoising.

[0075] In practical applications, to achieve accuracy and efficiency in denoising and motion detection, the motion processing model can be trained offline in advance. In one optional implementation method provided in this embodiment, the motion processing model is trained in the following way:

[0076] The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; optionally, the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence.

[0077] A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data.

[0078] The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

[0079] Optionally, the action frame sequence samples are obtained by sampling from the training sample set according to preset sampling parameters. In one optional implementation of this embodiment, the training sample set is constructed as follows:

[0080] Sample a preset action frame sequence from the action frame sequence pool, and determine the target action frame from the preset action frame sequence;

[0081] The target action frame is denoised according to each denoising parameter, and the multiple action frame sequences obtained by the denoising process are used as multiple action frame sequence samples. The training sample set is constructed based on the multiple action frame sequence samples.

[0082] In the process of adding noise to the target action frame according to each noise parameter, the target action frame can be noise-added according to each of the multiple noise parameters in a preset action frame sequence, obtaining multiple action frame sequences corresponding to multiple noise parameters as multiple action frame sequence samples. The noise parameter can be a parameter representing the amount of noise added. The preset action frame sequence includes a pre-stored continuous action frame sequence, such as a continuous dance action frame sequence or a continuous basketball action frame sequence. The target action frame includes any one or more preset action frames in the preset action frame sequence. Optionally, the target action frame is the start action frame and the end action frame in the preset action frame sequence.

[0083] Specifically, the process of inputting action frame sequence samples into the model to be trained for action frame denoising and action detection to obtain predicted action frame sequences and predicted action interaction data is similar to the process described above of inputting the action frame sequence to be processed and action feature parameters into the action processing model for action frame denoising and action detection to obtain candidate action frame sequences and action interaction data. The process of inputting action frame sequence samples into the model to be trained for action frame denoising and action detection to obtain predicted action frame sequences and predicted action interaction data can be found in the relevant content regarding inputting the action frame sequence to be processed and action feature parameters into the action processing model for action frame denoising and action detection to obtain candidate action frame sequences and action interaction data, and will not be repeated here. The target action frame here, during execution, is similar to the initial action frame sequence in the action frame sequence to be processed described above.

[0084] In the specific execution process, in order to improve the comprehensiveness and diversity of training loss, thereby improving the training effect of model training and the model accuracy of action processing model, in an optional implementation method provided in this embodiment, the following operation is performed during the calculation of the first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence:

[0085] The feature loss is calculated based on the part features of the predicted action frame and the part features of the target action frame. The position loss is calculated based on the position data of the target part in the predicted action frame and the position data of the target part in the target action frame. The motion change loss is calculated based on the previous and next action frames in the predicted action frame and the previous and next action frames in the target action frame.

[0086] In this process, after adding noise to the target action frame in the preset action frame sequence, a noisy action frame can be obtained. After inputting the action frame sequence sample into the model to be trained for action frame denoising and action detection, a predicted action frame sequence containing the target predicted action frame and predicted action interaction data can be obtained. That is, after denoising and action detection of the noisy action frame, the target predicted action frame is obtained.

[0087] The location features of the target predicted action frame include the rotation data features of the joints, the action interaction features between the toes and the ground, and / or the action interaction features between the heels and the ground; the location features of the target action frame also include the rotation data features of the joints, the action interaction features between the toes and the ground, and / or the action interaction features between the heels and the ground. The location features can be represented in the form of a feature matrix. For example, the location features may include the features of 24 joints, and the features of each joint can be represented by 3 rotation data. The location features may also include 4 action interaction features between the feet and the ground. When there are 3 target predicted action frames and 3 target action frames, the location features of the target predicted action frame and the location features of the target action frame can be represented as a feature vector [(24*3+4)*3].

[0088] Specifically, in the process of calculating the feature loss based on the part features of the predicted target action frame and the part features of the target action frame, the feature difference between the part features of the predicted target action frame and the part features of the target action frame can be calculated, and the feature loss can be calculated based on the feature difference. In the process of calculating the position loss based on the position data of the target part in the predicted target action frame and the position data of the target part in the target action frame, the rotation data of the target part in the predicted target action frame can be converted into position data of the target part, and the rotation data of the target part in the target action frame can be converted into position data of the target part. The position difference between the position data of the target part in the predicted target action frame and the position data of the target part in the target action frame can be calculated, and the position loss can be calculated based on the position difference. Here, the target part includes a joint, the rotation data of the target part includes the rotation angle or degree of rotation of the joint, and the position data of the target part includes the spatial coordinate position data of the joint.

[0089] In calculating the motion change loss based on the preceding and following action frames in the target predicted action frame and the preceding and following action frames in the target action frame, the motion change loss can be calculated based on the key feature characteristics of the preceding and following action frames in the target action frame, as well as the key feature characteristics of the preceding and following action frames in the target predicted action frame. More specifically, a first difference between the key feature characteristics of the preceding and following action frames in the target action frame and a second difference between the key feature characteristics of the preceding and following action frames in the target predicted action frame can be calculated respectively, and the motion change loss can be calculated based on the first and second differences. In this way, the target predicted action frame output by the model under training is made closer to the target action frame through feature loss, so that the model under training can learn to recover the target action frame from noisy action frames. The position loss ensures that the position of the target part in the target predicted action frame is consistent with the position of the target part in the target action frame. The motion change loss ensures that the change amplitude of the preceding and following frames in the target predicted action frame is consistent with the change amplitude of the preceding and following frames in the target action frame.

[0090] In addition, during the calculation of the first loss based on the target predicted action frame in the sequence of reference action frames and predicted action frames, one or more of the feature loss, position loss and action change loss can also be calculated.

[0091] In practical applications, target prediction action frames may exhibit object slippage. Therefore, to ensure the smoothness and continuity of the target prediction action frames output by the model under training, a second loss can be introduced to reduce slippage between consecutive frames. In one optional implementation of this embodiment, the following operation is performed during the calculation of the second loss based on the target prediction action frames and the predicted action interaction data:

[0092] The rotation data of the key parts in the previous action frame and the rotation data of the key parts in the next action frame in the target prediction action frame are converted into the first position data of the key parts in the previous action frame and the second position data of the key parts in the next action frame.

[0093] The second loss is calculated based on the first location data, the second location data, and the predicted action interaction data.

[0094] Specifically, in the process of calculating the second loss based on the first position data, the second position data, and the predicted action interaction data, the position difference between the second position data and the first position data can be calculated, and the second loss can be calculated based on the position difference and the action interaction state between the key parts in the previous action frame in the target predicted action frame and the target position.

[0095] For example, the first loss function for calculating feature loss is:

[0096]

[0097] Among them, L ddpm represents the feature loss, where x represents the part features of the target action frame. The part features representing the predicted action frame of the target. Representative to Find the norm. represent The square of the norm, E x，t represent The average of the squares of the norms;

[0098] The second loss function for calculating position loss is:

[0099]

[0100] Among them, L pos Represents the position loss; N represents each predicted action frame in the target predicted action frame, N = 1, 2, 3, ...; x i FK(x) represents the rotation data of 24 joints in each target action frame. iThe symbol represents the position data of the 24 joints in each target action frame, obtained by converting the rotation data of the 24 joints in each target action frame. This represents the rotation data of 24 joints in the predicted motion frame for each target. This represents the position data of the 24 joints in each target's predicted action frame, obtained by converting the rotation data of the 24 joints in each target's predicted action frame. represent The square of the norm;

[0101] The third loss function for calculating the loss due to changes in motion is:

[0102]

[0103] Among them, L v Represents the loss due to changes in action; x i+1 With x i These represent the next action frame and the previous action frame in the target action frame, respectively. and These represent the previous and next action frames in the target prediction action frames, respectively; ||(x i+1 -x i )-(xi+1--xi)2 represents the square of the norm of [xi+1-xi-(xi+1-xi)].

[0104] The fourth loss function for calculating the second loss is:

[0105]

[0106] Among them, L c This represents the second loss. This represents the rotational data of key parts (left toe, left heel, right toe, right heel) in the previous action frame of the target prediction action frame. This represents the position data of the key parts in the previous action frame, obtained by converting the rotation data of the key parts in the previous action frame. This represents the rotational data of key parts (left toe, left heel, right toe, right heel) in the next action frame of the target prediction action frame. This represents the position data of the key parts in the next action frame, obtained by converting the rotation data of the key parts in the next action frame. This represents the interaction state between the key parts of the previous action frame and the target location in the predicted action frame. It can be 0 or 1. When the value is 0, it indicates that the interaction state between the key part of the previous action frame in the target prediction action frame and the target position is in a contact state. When the value is 1, it means that the action interaction state between the key part of the previous action frame in the target prediction action frame and the target position is separated.

[0107] Having obtained the first and second losses, the parameters of the model to be trained can be adjusted based on these losses to obtain the action processing model. During the parameter adjustment process, the following operations can be performed: calculate the training loss based on the first and second losses, and then adjust the parameters of the model to be trained based on the training loss. Specifically, the training loss can be calculated based on the feature loss, the position loss and its position weights, the action change loss and its action change weights, the second loss and its weights, and the parameters of the model to be trained can be adjusted based on this training loss. The second loss can be the action interaction loss.

[0108] For example, the loss function of the model to be trained is:

[0109] L = L ddpm +λ pos L pos +λ v L v +λ c L c

[0110] Where, λ pos The location weights representing the location loss; λ v The weight of the action change that represents the loss due to action change; λ c The weight representing the second loss.

[0111] Referring to the model training process described above, the parameters of the model to be trained are iteratively adjusted until the loss function of the model to be trained converges, thus obtaining the action processing model.

[0112] Step S208: Perform motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence.

[0113] The above-mentioned motion processing model uses the action frame sequence to be processed and motion feature parameters to perform denoising and motion detection processing on the action frame sequence to be processed, obtaining candidate action frame sequences and motion interaction data. In this step, in order to further improve the motion smoothness of the final target action frame sequence, making each action frame in the target action frame sequence closer to the real action or posture, and improving the realism and effectiveness of the target action frame sequence, motion correction processing can be performed on the candidate action frame sequence based on the motion interaction data to obtain the target action frame sequence. For example Figure 3 The target action frame sequence shown is as follows. Figure 3 (2) and Figure 3 (3) Initialize the corrected action frame sequence corresponding to the action frame sequence.

[0114] In practical applications, the interaction state between key parts and the target position in each candidate action frame of the candidate action frame sequence may be in a contact state, but the key parts and the target position are not actually in contact, resulting in the object's movement not being smooth and fluid, and the phenomenon of slippage occurring, and the lack of continuity between the candidate action frames. To address this, in order to improve the continuity of the movement of each candidate action frame in the candidate action frame sequence and eliminate slippage, in an optional implementation of this embodiment, the following operation is performed during the process of performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain the target action frame sequence:

[0115] Determine the action interaction state between key parts and target positions in each candidate action frame of the action interaction data;

[0116] Based on the action interaction state, the key parts in each candidate action frame are corrected to obtain the target action frame sequence.

[0117] In one optional implementation of this embodiment, during the process of correcting the position of key parts in each candidate action frame according to the action interaction state, the following operations are performed:

[0118] If the action interaction state is a contact state, determine whether the key parts in each candidate action frame are at the target position;

[0119] If the target position is not reached, the key parts in each candidate action frame are locked at the target position;

[0120] If the target location is reached, no action needs to be taken.

[0121] In addition, if the action interaction state is a separation state, it is determined whether the key parts in each candidate action frame are separated from the target position; if they are separated from the target position, no processing is required; if they are in contact with the target position, the key parts locked to the target position are de-locked.

[0122] In addition, during the process of performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence, the candidate motion frame sequence can also be performed on the motion correction processing based on the motion interaction data using inverse kinematics to obtain the target motion frame sequence; wherein, inverse kinematics can be IK (Inverse Kinematics).

[0123] In addition to the above-described implementation method of performing motion correction processing on all candidate motion frames in the candidate motion frame sequence, in order to improve the targeting and effectiveness of motion correction, as well as its convenience and efficiency, motion correction processing can be performed only on the noise-reduced motion frame sequence in the candidate motion frame sequence. Specifically, in the process of performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence, the motion interaction state between the key parts in the noise-reduced motion frame sequence and the target position can also be determined, and the position correction processing of each motion frame in the noise-reduced motion frame sequence can be performed based on the determined motion interaction state to obtain the target motion frame sequence containing the corrected motion frame sequence; the position correction processing here is similar to the above implementation process.

[0124] In practical applications, the variation amplitudes between adjacent action frames in the target action frame sequence are not necessarily the same, and the difference in variation amplitudes may even be large. In this case, the smoothness of the target action frame sequence is still low and does not quite meet the conditions of a real action action frame sequence. To address this, in an optional implementation of this embodiment, after performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain the target action frame sequence, the following operations are also performed:

[0125] Calculate the feature difference of the part features between every two adjacent action frames in the target action frame sequence;

[0126] Based on the feature difference, the smoothing parameters of each action frame in the target action frame sequence are determined, and the action frames are smoothed according to the smoothing parameters to obtain the action frame sequence.

[0127] The location features include feature matrices characterizing joint and / or foot features, such as features characterizing rotational data of 24 joints and feature matrices characterizing the interaction state of key locations with the target position. The smoothing parameters include parameters for adjusting the location features of each action frame in the target action frame sequence.

[0128] Specifically, the feature difference between the part features of the next action frame and the part features of the previous action frame in each adjacent action frame of the target action frame sequence can be calculated. Based on the feature difference, the smoothing parameter of each action frame in the target action frame sequence is determined, and the action frames are smoothed based on the smoothing parameter to obtain the action frame sequence. In the process of smoothing the action frames according to the smoothing parameter to obtain the action frame sequence, the part features of each action frame can be adjusted according to the smoothing parameter to obtain the action frame sequence. In this way, the feature difference of the part features between each action frame in the action frame sequence can be as close as possible or the same, so as to achieve high smoothness and high coherence of the action frame sequence.

[0129] In summary, the one or more motion processing methods provided in this embodiment first obtain a start motion frame and a stop motion frame, sample a preset number of random numbers from a Gaussian distribution, and generate an initial motion frame sequence based on the sampled random numbers. This initial motion frame sequence is then filled into the middle padding positions of the start and stop motion frames to obtain a sequence of motion frames to be processed. Next, the sequence of motion frames to be processed and motion feature parameters are input into a motion processing model for denoising and motion detection, resulting in a candidate motion frame sequence and motion interaction data. Finally, the motion interaction state between key parts and target positions in each candidate motion frame in the motion interaction data is determined, and the key parts in each candidate motion frame are processed according to the motion interaction state. Position correction processing is performed on the parts to obtain the target action frame sequence. Finally, the feature difference of the part features of each adjacent action frame in the target action frame sequence is calculated. Based on the feature difference, the smoothing parameters of each action frame in the target action frame sequence are determined, and the action frames are smoothed according to the smoothing parameters to obtain the action frame sequence. In this way, the discontinuity between action frames in the target action frame sequence is avoided through position correction processing, and the smoothness and naturalness between action frames in the target action frame sequence are improved. At the same time, the diversity and flexibility of the target action frame sequence are realized through action feature parameters, making the action frames in the target action frame sequence closer to real action frames, and improving the effectiveness and realism of the target action frame sequence.

[0130] The following description uses the application of an action processing method provided in this embodiment in an action frame scene as an example to further illustrate the action processing method provided in this embodiment. See [link to documentation]. Figure 4 The motion processing method applied to motion frame scenes includes the following steps.

[0131] Step S402: Obtain the start action frame and the end action frame.

[0132] Step S404: Sample a preset number of random numbers from a Gaussian distribution and generate an initialization action frame based on the sampled random numbers.

[0133] Step S406: Fill the initialization action frame into the middle padding position between the start action frame and the end action frame to obtain the sequence of action frames to be processed.

[0134] Step S408: Input the action frame sequence to be processed and the action feature parameters into the action processing model to perform denoising processing and action detection processing of the action frames, and obtain candidate action frame sequences and action interaction data.

[0135] Optionally, the denoising process of the action frame includes: performing noise prediction processing on the initial action frame sequence in the action frame sequence to be processed based on action feature parameters, the start action frame, and the end action frame; and performing noise removal processing on the initial action frame sequence according to the noise prediction result to obtain a candidate action frame sequence containing the noise-removed action frame sequence.

[0136] Optionally, the action detection process includes: detecting whether the positional deviation of key parts between a candidate action frame and an adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or detecting whether the distance between the key parts in the candidate action frame and the target position is less than a preset distance threshold.

[0137] If the position deviation is less than a preset deviation threshold and / or the distance is less than a preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a contact state, and action interaction data is constructed based on the contact state, the key part, and the target position.

[0138] Step S410: Determine the action interaction state between the foot and the ground position in each candidate action frame of the action interaction data.

[0139] Step S412: Based on the action interaction state, perform position correction processing on the foot part in each candidate action frame to obtain the target action frame sequence.

[0140] Step S414: Calculate the feature difference of the part features of every two adjacent action frames in the target action frame sequence.

[0141] Optionally, the site features include a feature matrix characterizing joint features and / or the motion interaction state of the foot with the ground.

[0142] Step S416: Determine the smoothing parameters of each action frame in the target action frame sequence based on the feature difference, and perform action smoothing processing on each action frame according to the smoothing parameters to obtain the action frame sequence.

[0143] One or more embodiments of a model training method provided in this specification are as follows:

[0144] Reference Figure 5 The model training method provided in this embodiment specifically includes steps S502 to S506.

[0145] Step S502: Input the action frame sequence samples into the model to be trained for action frame denoising and action detection processing to obtain the predicted action frame sequence and predicted action interaction data.

[0146] The model training method provided in this embodiment, including the process of training the model to obtain the action processing model, and the process of denoising and detecting action frames through the action processing model to obtain the target action frame sequence, can refer to the relevant content in the action processing method provided in the above embodiment.

[0147] Optionally, the action frame sequence sample is obtained by adding noise to the target action frame in the preset action frame sequence.

[0148] Optionally, the action frame sequence samples are obtained by sampling from the training sample set according to preset sampling parameters. In one optional implementation of this embodiment, the training sample set is constructed as follows:

[0149] Sample a preset action frame sequence from the action frame sequence pool, and determine the target action frame from the preset action frame sequence;

[0150] The target action frame is denoised according to each denoising parameter, and the multiple action frame sequences obtained by the denoising process are used as multiple action frame sequence samples. The training sample set is constructed based on the multiple action frame sequence samples.

[0151] In the process of adding noise to the target action frame according to each noise parameter, the target action frame can be noise-added according to each of the multiple noise parameters to obtain multiple sets of action frames corresponding to the multiple noise parameters. The noise parameter can be a parameter representing the amount of noise added. The preset action frame sequence includes a pre-stored continuous action frame sequence, such as a continuous dance action frame sequence or a continuous basketball action frame sequence. The target action frame includes any one or more preset action frames in the preset action frame sequence. Optionally, the target action frame is the start action frame and the end action frame in the preset action frame sequence.

[0152] Specifically, the process of inputting action frame sequence samples into the model to be trained for action frame denoising and action detection to obtain predicted action frame sequences and predicted action interaction data is similar to the process described above of inputting the action frame sequence to be processed and action feature parameters into the action processing model for action frame denoising and action detection to obtain candidate action frame sequences and action interaction data. The process of inputting action frame sequence samples into the model to be trained for action frame denoising and action detection to obtain predicted action frame sequences and predicted action interaction data can be found in the relevant content regarding inputting the action frame sequence to be processed and action feature parameters into the action processing model for action frame denoising and action detection to obtain candidate action frame sequences and action interaction data, and will not be repeated here. The target action frame here, during execution, is similar to the initial action frame sequence in the action frame sequence to be processed described above.

[0153] Step S504: Calculate a first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence, and calculate a second loss based on the target predicted action frame and the predicted action interaction data.

[0154] In the specific execution process, in order to improve the comprehensiveness and diversity of training loss, thereby improving the training effect of model training and the model accuracy of action processing model, in an optional implementation method provided in this embodiment, the following operation is performed during the calculation of the first loss based on the target predicted action frame in the sequence of baseline action frames and predicted action frames:

[0155] The feature loss is calculated based on the part features of the predicted action frame and the part features of the target action frame. The position loss is calculated based on the position data of the target part in the predicted action frame and the position data of the target part in the target action frame. The motion change loss is calculated based on the previous and next action frames in the predicted action frame and the previous and next action frames in the target action frame.

[0156] In this process, after adding noise to the target action frame in the preset action frame sequence, a noisy action frame can be obtained. After inputting the action frame sequence sample into the model to be trained for action frame denoising and action detection, a predicted action frame sequence containing the target predicted action frame and predicted action interaction data can be obtained. That is, after denoising and action detection of the noisy action frame, the target predicted action frame is obtained.

[0157] The location features of the target predicted action frame include joint features, toe-to-ground interaction features, and / or heel-to-ground interaction features; the location features of the target action frame also include joint features, toe-to-ground interaction features, and / or heel-to-ground interaction features. Location features can be represented using a feature matrix. For example, location features may include features of 24 joints, each joint feature can be represented using 3 rotation data points, and location features may also include 4 foot-to-ground interaction features. When both the target predicted action frame and the target action frame have 3 frames, the location features of the target predicted action frame and the target action frame can be represented as a feature vector [(24*3+4)*3].

[0158] Specifically, in the process of calculating the feature loss based on the part features of the predicted target action frame and the part features of the target action frame, the feature difference between the part features of the predicted target action frame and the part features of the target action frame can be calculated, and the feature loss can be calculated based on the feature difference. In the process of calculating the position loss based on the position data of the target part in the predicted target action frame and the position data of the target part in the target action frame, the rotation data of the target part in the predicted target action frame can be converted into position data of the target part, and the rotation data of the target part in the target action frame can be converted into position data of the target part. The position difference between the position data of the target part in the predicted target action frame and the position data of the target part in the target action frame can be calculated, and the position loss can be calculated based on the position difference. Here, the target part includes a joint, the rotation data of the target part includes the rotation angle or degree of rotation of the joint, and the position data of the target part includes the spatial coordinate position data of the joint.

[0159] In calculating the motion change loss based on the preceding and following action frames in the target predicted action frame and the preceding and following action frames in the target action frame, the motion change loss can be calculated based on the key feature characteristics of the preceding and following action frames in the target action frame, as well as the key feature characteristics of the preceding and following action frames in the target predicted action frame. More specifically, a first difference between the key feature characteristics of the preceding and following action frames in the target action frame and a second difference between the key feature characteristics of the preceding and following action frames in the target predicted action frame can be calculated respectively, and the motion change loss can be calculated based on the first and second differences. In this way, the target predicted action frame output by the model under training is made closer to the target action frame through feature loss, so that the model under training can learn to recover the target action frame from noisy action frames. The position loss ensures that the position of the target part in the target predicted action frame is consistent with the position of the target part in the target action frame. The motion change loss ensures that the change amplitude of the preceding and following frames in the target predicted action frame is consistent with the change amplitude of the preceding and following frames in the target action frame.

[0160] In addition, during the calculation of the first loss based on the target predicted action frame in the sequence of reference action frames and predicted action frames, one or more of the feature loss, position loss and action change loss can also be calculated.

[0161] In practical applications, target prediction action frames may exhibit object slippage. Therefore, to ensure the smoothness and continuity of the target prediction action frames output by the model under training, a second loss can be introduced to reduce slippage between consecutive frames. In one optional implementation of this embodiment, the following operation is performed during the calculation of the second loss based on the target prediction action frames and the predicted action interaction data:

[0162] The rotation data of the key parts in the previous action frame and the rotation data of the key parts in the next action frame in the target prediction action frame are converted into the first position data of the key parts in the previous action frame and the second position data of the key parts in the next action frame.

[0163] The second loss is calculated based on the first location data, the second location data, and the predicted action interaction data.

[0164] Specifically, in the process of calculating the second loss based on the first position data, the second position data, and the predicted action interaction data, the position difference between the second position data and the first position data can be calculated, and the second loss can be calculated based on the position difference and the action interaction state between the key parts in the previous action frame in the target predicted action frame and the target position.

[0165] For example, the first loss function for calculating feature loss is:

[0166]

[0167] Among them, L ddpm represents the feature loss, where x represents the part features of the target action frame. The part features representing the predicted action frame of the target. Representative to Find the norm. represent The square of the norm, E x，t represent The average of the squares of the norms;

[0168] The second loss function for calculating position loss is:

[0169]

[0170] Among them, L pos Represents the position loss; N represents each predicted action frame in the target predicted action frame, N = 1, 2, 3, ...; x i FK(x) represents the rotation data of 24 joints in each target action frame. i The symbol represents the position data of the 24 joints in each target action frame, obtained by converting the rotation data of the 24 joints in each target action frame. This represents the rotation data of 24 joints in the predicted motion frame for each target. This represents the position data of the 24 joints in each target's predicted action frame, obtained by converting the rotation data of the 24 joints in each target's predicted action frame. represent The square of the norm;

[0171] The third loss function for calculating the loss due to changes in motion is:

[0172]

[0173] Among them, L v Represents the loss due to changes in action; x i+1 With x i These represent the next action frame and the previous action frame in the target action frame, respectively. and These represent the previous and next action frames in the target prediction action frames, respectively; ||(x i+1 -x i )-(xi+1-xi)2 represents the square of the norm of [xi+1-xi-(xi+1-xi)].

[0174] The fourth loss function for calculating the second loss is:

[0175]

[0176] Among them, L c This represents the second loss. This represents the rotational data of key parts (left toe, left heel, right toe, right heel) in the previous action frame of the target prediction action frame. This represents the position data of the key parts in the previous action frame, obtained by converting the rotation data of the key parts in the previous action frame. This represents the rotational data of key parts (left toe, left heel, right toe, right heel) in the next action frame of the target prediction action frame. This represents the position data of the key parts in the next action frame, obtained by converting the rotation data of the key parts in the next action frame. This represents the interaction state between the key parts of the previous action frame and the target location in the predicted action frame. It can be 0 or 1. When the value is 0, it indicates that the interaction state between the key part of the previous action frame in the target prediction action frame and the target position is in a contact state. When the value is 1, it means that the action interaction state between the key part of the previous action frame in the target prediction action frame and the target position is separated.

[0177] Step S506: Adjust the parameters of the model to be trained based on the first loss and the second loss to obtain the action processing model.

[0178] The first and second losses are obtained as described above. Based on these, the parameters of the model to be trained can be adjusted according to the first and second losses to obtain the action processing model. During the parameter adjustment process based on the first and second losses, the following operations can be performed: calculate the training loss based on the first and second losses, and adjust the parameters of the model to be trained based on the training loss. Specifically, the training loss can be calculated based on the feature loss, position loss and its position weights, action change loss and its action change weights, the second loss and its weights, and the parameters of the model to be trained can be adjusted based on the training loss. The second loss can be the action interaction loss. Optionally, the action processing model is used to perform denoising and action detection processing on the action frame sequence to be processed and action feature parameters to obtain candidate action frame sequences and action interaction data; the action interaction data is used to perform action correction processing on the candidate action frame sequences to obtain the target action frame sequence; the action frame sequence to be processed is obtained after filling the initial action frame sequence into the baseline action frame sequence. The denoising, action detection, and action correction processing here can refer to the relevant content in the action processing method provided in the above embodiments.

[0179] For example, the loss function of the model to be trained is:

[0180] L = L ddpm +λ pos L pos +λ v L v +λ c L c

[0181] Where, λ pos The location weights representing the location loss; λ v The weight of the action change that represents the loss due to action change; λ c The weight representing the second loss.

[0182] Referring to the model training process described above, the parameters of the model to be trained are iteratively adjusted until the loss function of the model to be trained converges, thus obtaining the action processing model.

[0183] After obtaining the motion processing model, denoising and motion detection processing of motion frames can be performed. Specifically, a baseline motion frame sequence can be obtained, and an initial motion frame sequence can be generated. The initial motion frame sequence is filled into the baseline motion frame sequence according to a preset filling method to obtain the motion frame sequence to be processed. The motion frame sequence to be processed and motion feature parameters are input into the motion processing model for denoising and motion detection processing of motion frames to obtain candidate motion frame sequences and motion interaction data. Based on the motion interaction data, motion correction processing is performed on the candidate motion frame sequences to obtain the target motion frame sequence.

[0184] The motion processing model described in this embodiment is used for denoising and motion detection of motion frames. It corrects the candidate motion frame sequence based on the obtained candidate motion frame sequence and motion interaction data to obtain the target motion frame sequence. In other words, the motion processing model generates a complete and continuous sequence of candidate motion frames based on a baseline motion frame sequence. For example, it generates a transition frame between the start and end motion frames. For instance, the motion processing model can be a diffusion model, such as a diffusion model with a DDPM (Denoising Diffusion Probabilistic Models) structure. DDPM is a generative model based on Markov noise diffusion and belongs to the diffusion model category. That is, the motion processing model described in this embodiment can adopt a diffusion model with a DDPM structure. This diffusion model is used for denoising and motion detection of motion frames to achieve the motion processing function. Furthermore, when the motion processing model is a diffusion model, other types of model structures can also be used besides the DDPM model structure.

[0185] The reference action frame sequence described in this embodiment refers to a sequence composed of one or more reference action frames. Optionally, the reference action frame sequence includes a start action frame and / or a stop action frame. Furthermore, the reference action frame sequence may also include a first action frame and / or a second action frame. In this embodiment, an action frame refers to an image frame used to characterize an object's action or posture, such as dancing, running, or playing ball. Additionally, the object's action or posture can be other types of actions or postures. The object here can be a real-world person, a virtual person, or other creatures capable of producing actions or postures, such as animals or marine life. Figure 3 The diagram shown is a schematic representation of the target action frame sequence. Figure 3 (1) It can be the starting action frame, Figure 3 (4) It can be a termination action frame.

[0186] It should be noted that the starting action frame here refers to the image frame that serves as the starting action or starting posture. Similarly, the ending action frame refers to the image frame that serves as the ending action or ending posture. The starting action frame can be one or more frames, and the ending action frame can also be one or more frames. In this embodiment, the action frame can be represented by a feature sequence of m joint parts. Each joint part is specifically represented by n degrees of freedom or rotation. The rotation can be the deflection angle or rotation angle of each joint part relative to the due south, due north, due west, or due east direction. In addition, the rotation can also be the deflection angle or rotation angle of each joint part relative to the target direction. For the key parts in each action frame, b can be used to represent the action interaction state.

[0187] The initialization action frame sequence refers to a sequence composed of one or more initialization action frames; the initialization action frames may be randomly generated noisy action frames. In an optional implementation of this embodiment, the following operations are performed during the process of obtaining the reference action frame sequence and generating the initialization action frame sequence:

[0188] Get the start and end action frames;

[0189] A preset number of random numbers are sampled from a Gaussian distribution, and the initialization action frame sequence is generated based on the sampled random numbers.

[0190] Specifically, the starting action frame and the ending action frame are obtained, and a baseline action frame sequence is constructed based on the starting action frame and the ending action frame. A preset number of random numbers are sampled from a Gaussian distribution, and an initialization action frame is constructed based on the sampled random numbers. An initialization action frame sequence is constructed based on the initialization action frame.

[0191] For example, to obtain the start and end action frames, 24*3 random numbers are sampled in a Gaussian distribution of 0-1. An initial action frame can be constructed based on the sampled 24*3 random numbers, and an initial action frame sequence can be constructed based on the constructed initial action frames. It should be noted that since each action frame can be represented by 3 rotation degrees of 24 joint parts, an initial action frame can be constructed by sampling 24*3 random numbers here.

[0192] In addition, during the process of acquiring the baseline action frame sequence and generating the initial action frame sequence, the start action frame or the end action frame can also be acquired; a preset number of random numbers are sampled from the Gaussian distribution, and the initial action frame sequence is generated based on the sampled random numbers.

[0193] In practical applications, there is often a need to generate transition frames. For example, given the start and end action frames of a dance, there is a need to generate a continuous and natural sequence of dance action frames based on the start and end action frames. To address this, a baseline action frame sequence can be obtained, and an initial action frame sequence can be generated.

[0194] In specific implementation, in addition to the methods of obtaining the baseline action frame sequence and generating the initial action frame sequence based on sampled random numbers, after obtaining the baseline action frame sequence, a preset number of random numbers can be sampled in a Gaussian distribution based on the baseline action frame sequence, and the initial action frames can be reconstructed based on the sampled random numbers. The initial action frame sequence can then be constructed based on the reconstructed initial action frames.

[0195] The preset filling method refers to the pre-set filling method for the base action frame sequence, such as filling the initialization action frame sequence to the middle filling position of the base action frame sequence.

[0196] In specific implementation, in order to obtain a continuous and natural target action frame sequence, an action frame sequence with the same number of frames as the target action frame sequence can be obtained first to form the basis for the output target action frame sequence. In an optional implementation provided in this embodiment, during the process of filling the initial action frame sequence into the base action frame sequence according to a preset filling method to obtain the action frame sequence to be processed, the following operations are performed:

[0197] The initial action frame sequence is filled into the middle padding position between the start action frame and the end action frame to obtain the action frame sequence to be processed.

[0198] Optionally, the intermediate padding bit refers to the middle position between the start action frame and the end action frame.

[0199] In addition to the aforementioned implementation where the padding is positioned between the start and end action frames, the aforementioned reference action frame sequence also includes a first action frame and a second action frame. In this case, the padding for filling the initialization action frame sequence can be positioned before the first action frame, after the second action frame, or between the first and second action frames. During the process of filling the initialization action frame sequence into the reference action frame sequence according to a preset padding method to obtain the action frame sequence to be processed, the following operations can be performed:

[0200] Determine the padding bits for each initialization action frame in the initialization action frame sequence within the base action frame sequence;

[0201] Each initialization action frame is filled into the padding bits in the reference action frame sequence to obtain the action frame sequence to be processed.

[0202] For example, in the initialization action frame sequence, the padding bit of the first initialization action frame in the first action frame and the second action frame is determined to be the first padding bit before the first action frame; the padding bit of the second initialization action frame in the first action frame and the second action frame is determined to be the second padding bit between the first action frame and the second action frame; and the padding bit of the third initialization action frame in the first action frame and the second action frame is determined to be the third padding bit after the second action frame. The first initialization action frame is filled to the first padding bit, the second initialization action frame is filled to the second padding bit, and the third initialization action frame is filled to the third padding bit to obtain the action frame sequence to be processed.

[0203] The action feature parameters refer to parameters that characterize the action features of an object. Optionally, the action feature parameters include action attribute parameters and / or action type parameters. The action attribute parameters include parameters that characterize action attributes, and the action type parameters include parameters that characterize action types. For example, the action attribute parameters are dancing, playing basketball, or running, and the action type parameters are cheerful or heavy movements. In addition, the action feature parameters, action attribute parameters, and action type parameters can also be other types of parameters.

[0204] The motion interaction data includes the motion interaction state between the key part and the target position, and the key part and / or the target position; optionally, the motion interaction state includes a contact state and / or a separation state. When the motion interaction state is in the contact state, it means that the key part and the target position are in contact. When the motion interaction state is in the separation state, it means that the key part and the target position are separated.

[0205] The key parts include key parts of the object; the key parts can be the foot parts, specifically the left toe, left heel, right toe and / or right heel; the target position refers to the target position in the action frame, such as the ground position in the action frame.

[0206] In practical implementation, since each initialization action frame in the initialization action frame sequence contains noise, in order to obtain an effective target action frame sequence, the initialization action frame sequence in the action frame sequence to be processed can be denoised. In an optional implementation provided in this embodiment, the denoising of the action frames is implemented in the following manner:

[0207] Based on the action feature parameters and the reference action frame sequence in the action frame sequence to be processed, noise prediction processing is performed on the initial action frame sequence in the action frame sequence to be processed.

[0208] The initial action frame sequence is subjected to noise reduction processing based on the noise prediction results to obtain the candidate action frame sequence containing the noise-reduced action frame sequence.

[0209] Furthermore, in order to further reduce the noise content of the noise-reduced action frame sequence and improve the effectiveness of the candidate action frame sequence, the initial action frame sequence can be denoised multiple times. In this case, the denoising of the action frames can be achieved in the following way:

[0210] Based on the action feature parameters and the baseline action frame sequence in the action frame sequence to be processed, noise prediction processing is performed on the initial action frame sequence in the action frame sequence to be processed.

[0211] The initial action frame sequence is noise-removed based on the noise prediction results, and the specific action frame sequence after noise removal is denoised based on the action feature parameters and the baseline action frame sequence.

[0212] Check if the number of denoising attempts exceeds the preset number of denoising attempts; if yes, obtain a candidate action frame sequence containing the noise-reduced action frame sequence; if no, return to perform denoising processing on the noise-reduced specific action frame sequence based on action feature parameters and the baseline action frame sequence.

[0213] After denoising the action frame sequence, to eliminate slippage in the denoised candidate action frame sequence and improve the smoothness and flexibility of the target action frame sequence, action detection processing can be performed on the candidate action frame sequence. This involves determining the position indicators of key parts in the candidate action frame sequence and detecting whether the position indicators of the key parts meet preset indicator conditions. If so, the action interaction state between the key parts and the target position in the candidate action frame is determined to be a contact state, and action interaction data is constructed based on the contact state, the key parts, and the target position. If not, the action interaction state between the key parts and the target position in the candidate action frame is determined to be a separation state, and action interaction data is constructed based on the separation state, the key parts, and the target position. Specifically, in an optional implementation method provided in this embodiment, action detection processing is performed in the following manner:

[0214] Detect whether the positional deviation of the key parts of the candidate action frame and the adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or, detect whether the distance between the key parts of the candidate action frame and the target position is less than a preset distance threshold;

[0215] If the position deviation is less than the preset deviation threshold and / or the distance is less than the preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a contact state, and the action interaction data is constructed based on the contact state, the key part and the target position;

[0216] If the position deviation is greater than or equal to the preset deviation threshold and / or the distance is greater than or equal to the preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a separated state, and action interaction data is constructed based on the separated state, the key part and the target position.

[0217] Optionally, the key parts include the heel and / or toe, the heel including the left heel and / or right heel, and the toe including the left toe and / or right toe. The positional deviation includes the difference between the position of the key parts in the candidate action frame and their position in adjacent action frames. The target position can be the ground position. In this embodiment, the candidate action frames in the candidate action frame sequence can be any action frame in the candidate action frame sequence.

[0218] It should be noted that the implementation process of motion detection here can also be the specific implementation process of detecting whether the position indicators of key parts meet the preset indicator conditions. If so, the motion interaction state between the key parts and the target position in the candidate motion frame is determined to be a contact state, and the motion interaction data is constructed based on the contact state, key parts and target position.

[0219] Furthermore, during the motion detection processing of motion frames, motion detection processing can also be performed only on the noise-reduced motion frame sequence in the candidate motion frame sequence. Specifically, the following operations can be performed: determine the position indicators of key parts in the noise-reduced motion frame sequence of the candidate motion frame sequence, and detect whether the position coordinates of the key parts meet the preset indicator conditions. If so, determine that the motion interaction state between the key parts and the target position in the noise-reduced motion frame sequence is a contact state, and construct motion interaction data based on the contact state, the key parts, and the target position. If not, determine that the motion interaction state between the key parts and the target position in the noise-reduced motion frame sequence is a separation state, and construct motion interaction data based on the separation state, the key parts, and the target position.

[0220] It should be noted that the process of detecting whether the position coordinates of the key parts meet the preset index conditions can be achieved by performing the following methods: detecting whether the position deviation of the key parts between the candidate action frame and the adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or detecting whether the distance between the key parts in the candidate action frame and the target position is less than a preset distance threshold.

[0221] In specific implementation, while inputting the sequence of action frames to be processed and action feature parameters into the action processing model for denoising and action detection to obtain candidate action frame sequences and action interaction data, the sequence of action frames to be processed and action feature parameters can also be input into the action processing model for denoising to obtain a denoised specific action frame sequence. The denoised specific action frame sequence and action feature parameters are then input into the action processing model for further denoising, and the number of denoising attempts is checked to see if it exceeds the preset number of denoising attempts. If so, a candidate action frame sequence is obtained, and the candidate action frame sequence is input into the action processing model for action detection to obtain action interaction data. If not, the process returns to performing the denoised specific action frame sequence and action feature parameters input into the action processing model for further denoising.

[0222] The above-mentioned motion processing model uses the action frame sequence to be processed and motion feature parameters to perform denoising and motion detection processing on the action frame sequence to be processed, obtaining candidate action frame sequences and motion interaction data. In this step, in order to further improve the motion smoothness of the final target action frame sequence, making each action frame in the target action frame sequence closer to the real action or posture, and improving the realism and effectiveness of the target action frame sequence, motion correction processing can be performed on the candidate action frame sequence based on the motion interaction data to obtain the target action frame sequence. For example Figure 3 As shown, Figure 3 (2) and Figure 3 (3) Initialize the corrected action frame sequence corresponding to the action frame sequence.

[0223] In practical applications, the interaction state between key parts and the target position in each candidate action frame of the candidate action frame sequence may be in a contact state, but the key parts and the target position are not actually in contact, resulting in the object's movement not being smooth and fluid, and the phenomenon of slippage occurring, and the lack of continuity between the candidate action frames. To address this, in order to improve the continuity of the movement of each candidate action frame in the candidate action frame sequence and eliminate slippage, in an optional implementation of this embodiment, the following operation is performed during the process of performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain the target action frame sequence:

[0224] Determine the action interaction state between key parts and target positions in each candidate action frame of the action interaction data;

[0225] Based on the action interaction state, the key parts in each candidate action frame are corrected to obtain the target action frame sequence.

[0226] In one optional implementation of this embodiment, during the process of correcting the position of key parts in each candidate action frame according to the action interaction state to obtain the target action frame sequence, the following operations are performed:

[0227] If the action interaction state is a contact state, determine whether the key parts in each candidate action frame are at the target position;

[0228] If the target position is not reached, the key parts in each candidate action frame are locked at the target position;

[0229] If the target location is reached, no action needs to be taken.

[0230] In addition, if the action interaction state is a separation state, it is determined whether the key parts in each candidate action frame are separated from the target position; if they are separated from the target position, no processing is required; if they are in contact with the target position, the key parts locked to the target position are de-locked.

[0231] Furthermore, in the process of performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence, the candidate motion frame sequence can also be performed on the motion correction processing based on the motion interaction data using inverse kinematics to obtain the target motion frame sequence; wherein, inverse kinematics can be IK (Inverse Kinematics).

[0232] In practical applications, the variation amplitudes between adjacent action frames in the target action frame sequence are not necessarily the same, and the difference in variation amplitudes may even be large. In this case, the smoothness of the target action frame sequence is still low and does not quite meet the conditions of a real action action frame sequence. To address this, in an optional implementation of this embodiment, after performing action correction processing on the candidate action frame sequence based on the action interaction data to obtain the target action frame sequence, the following operations are also performed:

[0233] Calculate the feature difference of the part features between every two adjacent action frames in the target action frame sequence;

[0234] Based on the feature difference, the smoothing parameters of each action frame in the target action frame sequence are determined, and the action frames are smoothed according to the smoothing parameters to obtain the action frame sequence.

[0235] The location features include feature matrices characterizing joint and / or foot features, such as features characterizing rotational data of 24 joints and feature matrices characterizing the interaction state of key locations with the target position. The smoothing parameters include parameters for adjusting the location features of each action frame in the target action frame sequence.

[0236] Specifically, the feature difference between the part features of the next action frame and the part features of the previous action frame in each adjacent action frame of the target action frame sequence can be calculated. Based on the feature difference, the smoothing parameter of each action frame in the target action frame sequence is determined, and the action frames are smoothed based on the smoothing parameter to obtain the action frame sequence. In the process of smoothing the action frames according to the smoothing parameter to obtain the action frame sequence, the part features of each action frame can be adjusted according to the smoothing parameter to obtain the action frame sequence. In this way, the feature difference of the part features between each action frame in the action frame sequence can be as close as possible or the same, so as to achieve high smoothness and high coherence of the action frame sequence.

[0237] The following description uses the application of a model training method provided in this embodiment in an action frame scene as an example to further illustrate the model training method provided in this embodiment. (See also...) Figure 6 The model training method applied to action frame scenarios includes the following steps.

[0238] Step S602: Input the action frame sequence samples into the model to be trained for action frame denoising and action detection processing to obtain the predicted action frame sequence and predicted action interaction data.

[0239] Optionally, the action frame sequence samples are obtained by adding noise to the target action frames in the preset action frame sequence.

[0240] Step S604: Calculate feature loss based on the part features of the target predicted action frame and the target action frame in the predicted action frame sequence; calculate position loss based on the position data of the target part in the target predicted action frame and the position data of the target part in the target action frame; and calculate action change loss based on the previous and next action frames in the target predicted action frame and the previous and next action frames in the target action frame.

[0241] Step S606: Convert the rotation data of the key parts of the previous action frame and the rotation data of the key parts of the next action frame in the target predicted action frame into the first position data of the key parts of the previous action frame and the second position data of the key parts of the next action frame. Calculate the action interaction loss based on the first position data, the second position data and the predicted action interaction data.

[0242] Step S608: Calculate the training loss based on feature loss, position loss, action change loss, and action interaction loss, and adjust the parameters of the model to be trained based on the training loss to obtain the action processing model.

[0243] The following is an embodiment of a motion processing device provided in this specification:

[0244] In the above embodiments, an action processing method is provided, and correspondingly, an action processing device is also provided, which will be described below with reference to the accompanying drawings.

[0245] Reference Figure 7 The diagram shows a schematic representation of an embodiment of an action processing device provided in this embodiment.

[0246] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0247] This embodiment provides an action processing device, including:

[0248] The action frame sequence acquisition module 702 is configured to acquire a baseline action frame sequence and generate an initial action frame sequence;

[0249] The action frame filling module 704 is configured to fill the initial action frame sequence into the reference action frame sequence according to a preset filling method to obtain the action frame sequence to be processed.

[0250] The denoising module 706 is configured to input the action frame sequence to be processed and the action feature parameters into the action processing model to perform denoising processing and action detection processing of the action frames, and obtain candidate action frame sequences and action interaction data.

[0251] The motion correction module 708 is configured to perform motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence.

[0252] The following is an embodiment of a model training device provided in this specification:

[0253] In the above embodiments, a model training method is provided, and correspondingly, a model training device is also provided, which will be described below with reference to the accompanying drawings.

[0254] Reference Figure 8 The diagram shows a schematic representation of an embodiment of a model training device provided in this embodiment.

[0255] Since the apparatus embodiments correspond to the method embodiments, the descriptions are relatively simple. For relevant parts, please refer to the corresponding descriptions of the method embodiments provided above. The apparatus embodiments described below are merely illustrative.

[0256] This embodiment provides a model training device, including:

[0257] The sample input module 802 is configured to input action frame sequence samples into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence.

[0258] The loss calculation module 804 is configured to calculate a first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence, and to calculate a second loss based on the target predicted action frame and the predicted action interaction data.

[0259] The parameter adjustment module 806 is configured to adjust the parameters of the model to be trained based on the first loss and the second loss to obtain an action processing model.

[0260] The following is an embodiment of a motion processing device provided in this specification:

[0261] Corresponding to the motion processing method described above, based on the same technical concept, one or more embodiments of this specification also provide a motion processing device for executing the motion processing method provided above. Figure 9 This is a schematic diagram of the structure of an action processing device provided for one or more embodiments of this specification.

[0262] This embodiment provides a motion processing device, including:

[0263] like Figure 9 As shown, motion processing devices can vary considerably due to differences in configuration or performance. They may include one or more processors 901 and memory 902, with memory 902 storing one or more application programs or data. Memory 902 can be temporary or persistent storage. The application programs stored in memory 902 may include one or more modules (not shown), each module including a series of computer-executable instructions from the motion processing device. Furthermore, processor 901 may be configured to communicate with memory 902, executing the series of computer-executable instructions stored in memory 902 on the motion processing device. The motion processing device may also include one or more power supplies 903, one or more wired or wireless network interfaces 904, one or more input / output interfaces 905, one or more keyboards 906, etc.

[0264] In one specific embodiment, the motion processing device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the motion processing device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0265] Obtain the baseline action frame sequence and generate the initialization action frame sequence;

[0266] The initial action frame sequence is filled into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed.

[0267] The action frame sequence to be processed and the action feature parameters are input into the action processing model to perform denoising and action detection processing of the action frames, so as to obtain candidate action frame sequences and action interaction data.

[0268] The candidate action frame sequence is processed by the action interaction data to obtain the target action frame sequence.

[0269] The following is an example of a model training device provided in this manual:

[0270] Corresponding to the model training method described above, based on the same technical concept, one or more embodiments of this specification also provide a model training device for executing the model training method provided above. Figure 10 This is a schematic diagram of the structure of a model training device provided in one or more embodiments of this specification.

[0271] This embodiment provides a model training device, including:

[0272] like Figure 10As shown, model training devices can vary significantly due to differences in configuration or performance. They may include one or more processors 1001 and memory 1002, with memory 1002 storing one or more application programs or data. Memory 1002 can be temporary or persistent storage. The application programs stored in memory 1002 may include one or more modules (not shown), each module including a series of computer-executable instructions from the model training device. Furthermore, processor 1001 may be configured to communicate with memory 1002, executing the series of computer-executable instructions stored in memory 1002 on the model training device. The model training device may also include one or more power supplies 1003, one or more wired or wireless network interfaces 1004, one or more input / output interfaces 1005, one or more keyboards 1006, etc.

[0273] In one specific embodiment, the model training device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer-executable instructions for the model training device, and is configured to be executed by one or more processors. The one or more programs include computer-executable instructions for performing the following:

[0274] The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence.

[0275] A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data.

[0276] The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

[0277] This specification provides an example of a storage medium as follows:

[0278] Corresponding to the action processing method described above, based on the same technical concept, one or more embodiments of this specification also provide a storage medium.

[0279] The storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0280] Obtain the baseline action frame sequence and generate the initialization action frame sequence;

[0281] The initial action frame sequence is filled into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed.

[0282] The action frame sequence to be processed and the action feature parameters are input into the action processing model to perform denoising and action detection processing of the action frames, so as to obtain candidate action frame sequences and action interaction data.

[0283] The candidate action frame sequence is processed by the action interaction data to obtain the target action frame sequence.

[0284] It should be noted that the embodiments of a storage medium described in this specification and the embodiments of an action processing method described in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can be referred to the implementation of the corresponding method described above, and the repeated parts will not be described again.

[0285] This specification provides an example of a storage medium as follows:

[0286] Corresponding to the model training method described above, based on the same technical concept, one or more embodiments of this specification also provide a storage medium.

[0287] The storage medium provided in this embodiment is used to store computer-executable instructions, which, when executed by a processor, implement the following process:

[0288] The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence.

[0289] A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data.

[0290] The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

[0291] It should be noted that the embodiments of a storage medium and the embodiments of a model training method in this specification are based on the same inventive concept. Therefore, the specific implementation of the embodiments can be found in the implementation of the corresponding methods described above, and the repeated parts will not be described again.

[0292] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, please refer to each other. Each embodiment focuses on describing the differences from other embodiments. For example, the device embodiment, equipment embodiment, and storage medium embodiment are all similar to the method embodiment, so the description is relatively simple. For reading the relevant content of the device embodiment, equipment embodiment, and storage medium embodiment, please refer to the description of the method embodiment.

[0293] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0294] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using a hardware physical module. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Comell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0295] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0296] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0297] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0298] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0299] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0300] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0301] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0302] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0303] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0304] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0305] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0306] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0307] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0308] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A motion processing method, comprising: Obtain the baseline action frame sequence and generate the initialization action frame sequence; The initial action frame sequence is filled into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed. The process involves inputting the action frame sequence to be processed and the action feature parameters into the action processing model for denoising and action detection, thereby obtaining candidate action frame sequences and action interaction data. This includes: inputting the action frame sequence to be processed and the action feature parameters into the action processing model for denoising, obtaining a denoised specific action frame sequence; inputting the denoised specific action frame sequence and the action feature parameters into the action processing model for further denoising; detecting whether the number of denoising iterations exceeds a preset number of iterations; if yes, outputting a candidate action frame sequence; inputting the candidate action frame sequence into the action processing model for action detection, thereby obtaining action interaction data; if no, returning to the previous step to perform the denoising operation of inputting the denoised specific action frame sequence and the action feature parameters into the action processing model. The training method of the action processing model includes: inputting action frame sequence samples into the model to be trained to obtain target predicted action frames and predicted action interaction data; jointly optimizing the model parameters based on the first loss of the target action frames and the target predicted action frames, and the second loss of the target predicted action frames and the predicted action interaction data, to obtain the action processing model. The motion interaction data includes the motion interaction state between the key parts and the target position, the key parts and / or the target position, and the motion interaction state includes contact state and / or separation state. The candidate action frame sequence is processed by the action interaction data to obtain the target action frame sequence.

2. The method according to claim 1, wherein the noise reduction processing of the motion frame includes: Based on the action feature parameters and the reference action frame sequence in the action frame sequence to be processed, noise prediction processing is performed on the initial action frame sequence in the action frame sequence to be processed. The initial action frame sequence is subjected to noise reduction processing based on the noise prediction results to obtain the candidate action frame sequence containing the noise-reduced action frame sequence.

3. The method according to claim 1, wherein the motion detection processing includes: Detect whether the positional deviation of the key parts of the candidate action frame and the adjacent action frame in the candidate action frame sequence is less than a preset deviation threshold, and / or, detect whether the distance between the key parts of the candidate action frame and the target position is less than a preset distance threshold; If the position deviation is less than the preset deviation threshold and / or the distance is less than the preset distance threshold, the action interaction state between the key part and the target position in the candidate action frame is determined to be a contact state, and the action interaction data is constructed based on the contact state, the key part and the target position.

4. The method according to claim 1, wherein obtaining the reference action frame sequence and generating the initialization action frame sequence comprises: Get the start and end action frames; A preset number of random numbers are sampled from a Gaussian distribution, and the initialization action frame sequence is generated based on the sampled random numbers.

5. The method according to claim 4, wherein filling the initial action frame sequence into the reference action frame sequence according to a preset filling method to obtain the action frame sequence to be processed includes: The initial action frame sequence is filled into the middle padding position between the start action frame and the end action frame to obtain the action frame sequence to be processed.

6. The method according to claim 1, wherein performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence comprises: Determine the action interaction state between key parts and target positions in each candidate action frame of the action interaction data; Based on the action interaction state, the key parts in each candidate action frame are corrected to obtain the target action frame sequence.

7. The method according to claim 6, wherein the step of performing position correction processing on key parts in each candidate action frame based on the action interaction state includes: If the action interaction state is a contact state, determine whether the key parts in each candidate action frame are at the target position; If the target position is not reached, the key parts in each candidate action frame are locked at the target position.

8. The method according to claim 1, further comprising, after performing the step of performing motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence: Calculate the feature difference of the part features of every two adjacent action frames in the target action frame sequence; Based on the feature difference, the smoothing parameters of each action frame in the target action frame sequence are determined, and the action frames are smoothed according to the smoothing parameters to obtain the action frame sequence.

9. The method according to claim 1, wherein the action processing model is trained in the following manner: The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence. A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

10. The method according to claim 9, wherein calculating the first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence comprises: The feature loss is calculated based on the part features of the predicted action frame and the part features of the target action frame. The position loss is calculated based on the position data of the target part in the predicted action frame and the position data of the target part in the target action frame. The motion change loss is calculated based on the previous and next action frames in the predicted action frame and the previous and next action frames in the target action frame.

11. The method according to claim 9, wherein calculating the second loss based on the target predicted action frame and the predicted action interaction data comprises: The rotation data of the key parts in the previous action frame and the rotation data of the key parts in the next action frame in the target prediction action frame are converted into the first position data of the key parts in the previous action frame and the second position data of the key parts in the next action frame. The second loss is calculated based on the first location data, the second location data, and the predicted action interaction data.

12. The method according to claim 9, wherein the action frame sequence sample is obtained by sampling from the training sample set according to preset sampling parameters; The training sample set is constructed using the following sampling method: Sample a preset action frame sequence from the action frame sequence pool, and determine the target action frame from the preset action frame sequence; The target action frame is denoised according to each denoising parameter, and the multiple action frame sequences obtained by the denoising process are used as multiple action frame sequence samples. The training sample set is constructed based on the multiple action frame sequence samples.

13. A model training method, comprising: The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence. A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data, including: Calculate the position difference between the second position data of the key part in the next action frame and the first position data of the key part in the previous action frame, and calculate the second loss based on the position difference and the action interaction state between the key part in the previous action frame and the target position in the target prediction action frame. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

14. The method of claim 13, wherein calculating the first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence comprises: The feature loss is calculated based on the part features of the predicted action frame and the part features of the target action frame. The position loss is calculated based on the position data of the target part in the predicted action frame and the position data of the target part in the target action frame. The motion change loss is calculated based on the previous and next action frames in the predicted action frame and the previous and next action frames in the target action frame.

15. The method according to claim 13, wherein calculating the second loss based on the target predicted action frame and the predicted action interaction data comprises: The rotation data of the key parts in the previous action frame and the rotation data of the key parts in the next action frame are converted into the first position data of the key parts in the previous action frame and the second position data of the key parts in the next action frame.

16. A motion processing device, comprising: The action frame sequence acquisition module is configured to acquire a baseline action frame sequence and generate an initial action frame sequence; An action frame filling module is configured to fill the initial action frame sequence into the reference action frame sequence according to a preset filling method to obtain the action frame sequence to be processed. The denoising module is configured to input the action frame sequence to be processed and action feature parameters into the action processing model for denoising and action detection processing of the action frames, and obtain candidate action frame sequences and action interaction data. The module includes: inputting the action frame sequence to be processed and action feature parameters into the action processing model for denoising of the action frames, obtaining a denoised specific action frame sequence; inputting the denoised specific action frame sequence and action feature parameters into the action processing model for denoising of the action frames; detecting whether the number of denoising operations exceeds a preset number of denoising operations; if yes, outputting a candidate action frame sequence; inputting the candidate action frame sequence into the action processing model for action detection processing of the action frames, and obtaining action interaction data; if no, returning to perform the operation of inputting the denoised specific action frame sequence and action feature parameters into the action processing model for denoising of the action frames. The training method of the action processing model includes: inputting action frame sequence samples into the model to be trained to obtain target predicted action frames and predicted action interaction data; jointly optimizing the model parameters based on the first loss of the target action frames and the target predicted action frames, and the second loss of the target predicted action frames and the predicted action interaction data, to obtain the action processing model. The motion interaction data includes the motion interaction state between the key parts and the target position, the key parts and / or the target position, and the motion interaction state includes contact state and / or separation state. The motion correction module is configured to perform motion correction processing on the candidate motion frame sequence based on the motion interaction data to obtain the target motion frame sequence.

17. A model training device, comprising: The sample input module is configured to input action frame sequence samples into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence. The loss calculation module is configured to calculate a first loss based on the target action frame and the target predicted action frame in the predicted action frame sequence, and to calculate a second loss based on the target predicted action frame and the predicted action interaction data. The loss calculation module is also configured to calculate the position difference between the second position data of the key part of the subsequent action frame and the first position data of the key part of the previous action frame, and to calculate the second loss based on the position difference and the action interaction state between the key part of the previous action frame and the target position in the target predicted action frame. The parameter adjustment module is configured to adjust the parameters of the model to be trained based on the first loss and the second loss to obtain an action processing model.

18. A motion processing device, comprising: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: Obtain the baseline action frame sequence and generate the initialization action frame sequence; The initial action frame sequence is filled into the reference action frame sequence according to the preset filling method to obtain the action frame sequence to be processed. The sequence of action frames to be processed and the action feature parameters are input into the action processing model for noise reduction of the action frames. The process involves motion detection and processing to obtain candidate motion frame sequences and motion interaction data, including: inputting the motion frame sequence to be processed and motion feature parameters into the motion processing model for motion frame denoising to obtain a denoised specific motion frame sequence; inputting the denoised specific motion frame sequence and motion feature parameters into the motion processing model for motion frame denoising; detecting whether the number of denoising operations exceeds a preset number of denoising operations; if so, outputting a candidate motion frame sequence; inputting the candidate motion frame sequence into the motion processing model for motion detection processing to obtain motion interaction data; if not, returning to perform the operation of inputting the denoised specific motion frame sequence and motion feature parameters into the motion processing model for motion frame denoising. The training method of the action processing model includes: inputting action frame sequence samples into the model to be trained to obtain target predicted action frames and predicted action interaction data; jointly optimizing the model parameters based on the first loss of the target action frames and the target predicted action frames, and the second loss of the target predicted action frames and the predicted action interaction data, to obtain the action processing model. The motion interaction data includes the motion interaction state between the key parts and the target position, the key parts and / or the target position, and the motion interaction state includes contact state and / or separation state. The candidate action frame sequence is processed by the action interaction data to obtain the target action frame sequence.

19. A model training device, comprising: A processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to: The action frame sequence samples are input into the model to be trained for action frame denoising and action detection processing to obtain predicted action frame sequences and predicted action interaction data; the action frame sequence samples are obtained after adding noise to the target action frames in the preset action frame sequence. A first loss is calculated based on the target action frame and the target predicted action frame in the predicted action frame sequence, and a second loss is calculated based on the target predicted action frame and the predicted action interaction data, including: Calculate the position difference between the second position data of the key part in the next action frame and the first position data of the key part in the previous action frame, and calculate the second loss based on the position difference and the action interaction state between the key part in the previous action frame and the target position in the target prediction action frame. The parameters of the model to be trained are adjusted based on the first loss and the second loss to obtain the action processing model.

Images