Data processing method, and training method and device of motion prediction model

Abstract

The application discloses a data processing method and a motion prediction model training method and device. The data processing method comprises the following steps: acquiring continuous action frames of at least one target object captured, and two or more motion capture marker points of the target object are marked in each action frame of the continuous action frames; acquiring initial feature data of the motion capture marker points in the continuous action frames according to the continuous action frames; inputting the initial feature data into a motion prediction model, and obtaining prediction feature data with inter-action frame relationship and intra-action frame relationship through a denoising network of the motion prediction model, and obtaining an action sequence of at least one motion capture marker point of the target object in the continuous action frames according to the prediction feature data by the motion prediction model; wherein the intra-action frame relationship comprises a correlation relationship between different motion capture marker points in each action frame, and the inter-action frame relationship comprises a correlation relationship between the same motion capture marker points in each action frame.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to data processing methods, training methods and apparatus for motion prediction models. Background Technology

[0002] Motion capture (MoCap) is a technology that uses specific external devices to record and reconstruct the motion of moving objects. In recent years, with the development of related technologies, optical motion capture technology based on computer vision principles has become a commonly used motion capture method in the industry, and it is widely used in the motion production processes of games, movies, and other related fields.

[0003] In practical applications, optical motion capture technology uses multiple high-speed cameras with different perspectives to simultaneously capture the actor's movements, and uses multi-view 3D reconstruction technology to calculate the 3D coordinates of markers attached to different parts of the actor's body. Finally, based on these 3D coordinates, the position and motion information of each marker of the actor are calculated.

[0004] However, during the motion capture data acquisition process, due to factors such as human occlusion of marker points, the accuracy of motion capture equipment, and the error of the algorithm itself, the coordinates of each marker point recorded by the optical motion capture equipment will inevitably have errors. Therefore, additional cleaning and noise reduction processing is required for the motion capture data.

[0005] Currently, noise reduction of motion capture data is mostly done manually. Although manual noise reduction of motion capture data is effective, it still suffers from high cost, huge workload, and cumbersome process. Summary of the Invention

[0006] This application provides a data processing method to address the problems of high cost, large workload, and cumbersome process in denoising motion capture data in existing technologies. This application also provides a training method and apparatus for a motion prediction model.

[0007] A data processing method, characterized in that it includes:

[0008] Acquire continuous motion frames of at least one captured target object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the target object;

[0009] Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames;

[0010] The initial feature data is input into the motion prediction model, and through the denoising network of the motion prediction model, predicted feature data containing inter-frame and intra-frame relationships of actions is obtained; and

[0011] The motion prediction model obtains the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the prediction feature data;

[0012] The intra-frame relationship includes the association between different motion capture markers in each motion frame, and the inter-frame relationship includes the association between the same motion capture markers in adjacent motion frames.

[0013] Optionally, obtaining the initial feature data of the motion capture marker in the continuous motion frames based on the continuous motion frames includes:

[0014] The initial feature data of the target object in the continuous action frames are obtained based on the number of consecutive action frames, the number of motion capture markers in each action frame, and the three-dimensional coordinates of each motion capture marker.

[0015] Optionally, the denoising network includes an inter-frame nonlocal module and an intra-frame nonlocal module, wherein the inter-frame nonlocal module is used to determine the inter-frame relationship of the action, and the intra-frame nonlocal module is used to determine the intra-frame relationship of the action.

[0016] Optionally, the step of inputting the initial feature data into the motion prediction model and obtaining predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model includes:

[0017] The initial feature data is input into the motion prediction model, and the initial feature data is processed by the intra-frame nonlocal module to obtain first feature data that has the correlation between different motion capture markers in each action frame.

[0018] The first feature data is processed by the inter-frame non-local module, and the predicted feature data is obtained based on the processing result. The predicted feature data includes the correlation between different motion capture markers in each action frame and the correlation between the same motion capture markers in adjacent action frames.

[0019] Optionally, the step of processing the initial feature data through the intra-frame non-local module to obtain first feature data containing the correlation between different motion capture marker points in each action frame includes:

[0020] The initial feature data is transposed to obtain the transposed embedded feature data.

[0021] The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels.

[0022] Multiply the transposed first or second intermediate feature with another intermediate feature to obtain the first feature data that has the correlation relationship between different motion capture marker points in the action frame.

[0023] Optionally, the step of processing the first feature data through the inter-frame non-local module and obtaining the predicted feature data based on the processing result includes:

[0024] The first feature data is compressed by the third and fourth compression networks of the inter-frame nonlocal module to obtain the third and fourth intermediate features that represent the image features of the continuous action frames using different channels.

[0025] The transposed third or fourth intermediate feature is multiplied by a target intermediate feature to obtain the predicted feature data.

[0026] Optionally, the step of inputting the initial feature data into the motion prediction model and obtaining predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model includes:

[0027] The initial feature data is input into the motion prediction model, and the initial feature data is processed by the inter-frame nonlocal module to obtain second feature data that has the correlation between the same motion capture marker points in adjacent action frames.

[0028] The second feature data is processed by the intra-frame non-local module, and the predicted feature data is obtained based on the processing result. The predicted feature data includes the correlation between the same motion capture markers in adjacent action frames and the correlation between different motion capture markers in each action frame.

[0029] Optionally, the denoising network further includes a fully connected layer, which is used to predict the action sequence of each marker point of the target object in the continuous action frames based on the predicted feature data.

[0030] This application also provides a method for training a motion prediction model, including:

[0031] Acquire continuous motion frames of at least one captured sample object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the sample object;

[0032] Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames;

[0033] The initial feature data is input into the motion prediction model, and the denoising network of the motion prediction model is used to obtain predicted feature data containing inter-frame and intra-frame relationships of actions; and

[0034] The motion prediction model obtains a first motion sequence of at least one motion capture marker of the sample object in the consecutive motion frames based on the prediction feature data; wherein, the intra-motion frame relationship includes the association relationship between different markers in each motion frame, and the inter-motion frame relationship includes the association relationship between the same markers in adjacent motion frames;

[0035] Obtain the actual motion sequence of the motion capture markers of the sample object in the continuous motion frames;

[0036] Based on the first motion sequence and the actual motion sequence, the model parameters of the motion prediction model are optimized to obtain a parameter-optimized motion prediction model.

[0037] Optionally, optimizing the parameters of the motion prediction model based on the first motion sequence and the actual motion sequence includes:

[0038] A loss function is constructed based on the difference between the first motion prediction sequence and the actual motion sequence, and the motion model is optimized based on the loss function.

[0039] This application also provides a data processing apparatus, including:

[0040] The first acquisition unit is used to acquire at least one continuous motion frame of a target object, wherein each motion frame of the continuous motion frame is marked with two or more motion capture markers of the target object.

[0041] The second acquisition unit is used to acquire the initial feature data of the motion capture marker in the continuous action frames based on the continuous action frames.

[0042] The model prediction unit is used to input the initial feature data into the motion prediction model, and obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the predicted feature data.

[0043] The intra-frame relationships include the associations between different joints in each action frame, and the inter-frame relationships include the associations between the same joints in adjacent action frames.

[0044] This application also provides a training device for a motion prediction model, characterized in that it includes:

[0045] The third acquisition unit is used to acquire at least one captured continuous motion frames of a sample object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the sample object.

[0046] The fourth acquisition unit is used to acquire the initial feature data of the motion capture marker in the continuous motion frames based on the continuous motion frames.

[0047] A sample prediction unit is configured to input the initial feature data into a motion prediction model, obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains a first action sequence of at least one motion capture marker of the sample object in the consecutive motion frames based on the predicted feature data; wherein the intra-frame relationship includes the association between different markers in each motion frame, and the inter-frame relationship includes the association between the same markers in adjacent motion frames;

[0048] The fifth acquisition unit is used to obtain the actual motion sequence of the motion capture marker points of the sample object in the continuous motion frames;

[0049] The model training unit is used to optimize the model parameters of the motion prediction model based on the first motion sequence and the actual motion sequence, so as to obtain a parameter-optimized motion prediction model.

[0050] This application also provides an electronic device, including:

[0051] processor;

[0052] A memory for storing a program of a method, which, when read and executed by a processor, performs any of the methods described above.

[0053] This application also provides a computer storage medium storing a computer program, which, when executed, performs any of the methods described above.

[0054] Compared with the prior art, this application has the following advantages:

[0055] The data processing method provided in this application, after obtaining the initial feature data of motion capture markers of the target object in continuous action frames, mines the correlation between different motion capture markers in each action frame and the correlation between the same motion capture markers in different action frames based on a motion prediction model. Then, combining the correlation between different motion capture markers in adjacent action frames and the correlation between the same motion capture markers in different action frames, it predicts the motion sequence of the target object's motion capture markers in continuous action frames. This method not only considers the spatial relationship between different motion capture markers in each action frame but also the temporal correlation of motion capture markers in different action frames, thus completing the prediction of the target object's motion process. It also achieves denoising processing of continuous action frames. This method reduces the denoising cost of motion capture data and improves the denoising efficiency of motion capture data. Attached Figure Description

[0056] Figure 1 A flowchart of a data processing method provided in an embodiment of this application;

[0057] Figure 2 A schematic diagram of motion capture marker points for a target object provided in another embodiment of this application;

[0058] Figure 3 This is a schematic diagram of intra-frame relationships for actions provided in another embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the inter-frame relationship of actions provided in another embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of an intra-frame non-local module provided in another embodiment of this application;

[0061] Figure 6 This is a schematic diagram of the structure of an inter-frame nonlocal module provided in another embodiment of this application;

[0062] Figure 7 This is a schematic diagram illustrating an application scenario of the data processing method provided in another embodiment of this application;

[0063] Figure 8 A flowchart illustrating a training method for a motion sequence prediction model provided in another embodiment of this application;

[0064] Figure 9 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application;

[0065] Figure 10 A schematic diagram of a training device for a motion prediction model provided in another embodiment of this application;

[0066] Figure 11This is a schematic diagram of an electronic device structure provided for another embodiment of this application. Detailed Implementation

[0067] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0068] This application provides a data processing method, a method and apparatus for training a motion prediction model, which will be described in detail in the following embodiments.

[0069] This application first provides a data processing method, please refer to... Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application, which includes the following steps S101 to S103.

[0070] Step S101: Obtain at least one captured continuous motion frames of a target object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the target object.

[0071] In this embodiment, the target object refers to the moving target tracked by the motion capture device. The essence of motion capture is to measure, track and record the motion trajectory of the target object in three-dimensional space.

[0072] The continuous motion frames of the target object are the motion frames to be corrected, which contain noisy data and are obtained by the motion capture device tracking and capturing the target object. The motion frames are marked with motion capture markers of the target object.

[0073] Please refer to Figure 2 This is a schematic diagram of motion capture markers for a target object provided in another embodiment of this application.

[0074] like Figure 2 As shown, Figure 2 The target object 201 includes a wearable motion capture device, and the motion capture markers set by the motion capture device are as follows:

[0075] The target object's head motion capture marker 1, chest motion capture marker 2, left shoulder motion capture marker 3, right shoulder motion capture marker 4, left elbow joint motion capture marker 5, right elbow joint motion capture marker 6, left hand motion capture marker 7, right hand motion capture marker 8, left hip joint motion capture marker 9, right hip joint motion capture marker 10, left knee joint motion capture marker 11, right knee joint motion capture marker 12, left foot motion capture marker 13, and right foot motion capture marker 14.

[0076] After determining the motion capture markers, the coordinates of the motion capture markers can be determined based on the sensors installed at the corresponding marker locations by the motion capture device.

[0077] Understandably, the above... Figure 2 The description is only for the purpose of facilitating understanding of the motion capture markers provided in this application, and is not intended to limit the form of existence of the motion capture markers provided in this application. In actual application, the motion capture markers can be set in different forms and in different quantities according to the actual situation, and this application does not impose any restrictions on this.

[0078] Step S102: Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames.

[0079] In this application, the initial feature data of the motion capture marker in the continuous motion frames refers to the three-dimensional coordinate sequence of the motion capture marker. In an optional embodiment of this application, the dimension of the three-dimensional coordinate sequence is T×M×3, where T represents the number of continuous motion frames, M represents the number of motion capture markers in each frame, and 3 represents the three-dimensional coordinates of the motion capture marker, that is, the position information of the motion capture marker in the preset three-dimensional coordinates (x, y, z).

[0080] Step S103: Input the initial feature data into the motion prediction model, and obtain the predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and obtain the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the predicted feature data.

[0081] The above step S103 is mainly used to denoise the initial feature data of the motion capture markers and determine the motion sequence of each motion capture marker in the continuous motion frames.

[0082] In the process of implementation and application, digital images are often affected by noise interference from imaging equipment and the external environment during digitization and transmission. Noise is one of the important causes of image interference, and an image may contain various kinds of noise in practical applications. In the embodiments of this application, the noise in the continuous motion frames refers to the error between the three-dimensional coordinates of the motion capture marker points of the target object and the actual three-dimensional coordinates of the motion capture marker points in the continuous motion frames.

[0083] In one existing technique, denoising of the motion capture markers relies on fully connected layers (FC) in a convolutional neural network. Specifically, the existing technique trains the fully connected layer by constructing several noisy and noiseless single-frame sample images as sample data, resulting in a fully connected layer used for denoising single-frame images, thereby achieving denoising of motion frames. The existing method uses simple, basic fully connected layers to build the neural network, processing the motion capture markers in each frame one by one, thus denoising continuous motion frames.

[0084] In machine learning, the fully connected layer acts as a "classifier" in a convolutional neural network. If the operations of convolutional layers, pooling layers, and activation function layers in a convolutional neural network are to map the original data to the hidden feature space, then the fully connected layer is to map the distributed feature representation learned by the convolutional neural network to the sample label space.

[0085] However, the existing technology of denoising the continuous action frames using a convolutional neural network does not take into account the spatial correlation between different marker points in each frame and the temporal correlation of the movement of the same marker points between different action frames, resulting in low denoising accuracy.

[0086] In view of the problems existing in the prior art, the motion prediction model provided in this application includes: intra-frame nonlocal modules and inter-frame nonlocal modules;

[0087] The intra-frame nonlocal module is used to determine the intra-frame relationship of the action, which refers to the correlation between different motion capture markers in the spatial dimension of each action frame; the inter-frame nonlocal module is used to determine the inter-frame relationship of the action, which refers to the association between the same markers in adjacent action frames.

[0088] To facilitate understanding of the intra-frame and inter-frame relationships of actions provided in the embodiments of this application, the following is combined with... Figure 3 and Figure 4 The intra-frame relationships and inter-frame relationships of the actions are described in detail respectively.

[0089] Please refer to Figure 3 , Figure 3 This is a schematic diagram of intra-frame relationships for another embodiment of this application.

[0090] Figure 3 It includes: target object 301 and motion capture markers 1 to 18 of target object 301.

[0091] The intra-frame relationship of the motion capture point refers to the relationship between motion capture point 1 and other motion capture points (motion capture point 2 to motion capture point 18), the relationship between motion capture point 2 and other motion capture points (motion capture point 1, motion capture point 3 to motion capture point 18), ..., the relationship between motion capture point 17 and other motion capture points (motion capture point 1 to motion capture point 16, motion capture point 18), and the relationship between motion capture point 18 and other motion capture points (motion capture point 1 to motion capture point 17).

[0092] For further details, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the inter-frame relationship of actions provided in another embodiment of this application.

[0093] Figure 4 The image shows the same target object 401 in consecutive motion frames 1 to 3, and motion capture markers 1 to 18 of the target object 401.

[0094] like Figure 4 As shown, the relationship between action frames refers to the relationship between motion capture marker 1 in action frame 1, motion capture marker 1 in action frame 2, and motion capture marker 1 in action frame 3; the relationship between motion capture marker 2 in action frame 1, motion capture marker 2 in action frame 2, and motion capture marker 2 in action frame 3; ...; the relationship between motion capture marker 18 in action frame 1, motion capture marker 18 in action frame 2, and motion capture marker 18 in action frame 3.

[0095] Before obtaining the inter-frame and intra-frame relationships of the action through the denoising network, the initial feature data must first be converted into feature data suitable for the input of the denoising network.

[0096] Specifically, the motion prediction model also includes a feature embedding module.

[0097] The feature embedding module uses the transposed initial feature data (i.e., the transposed three-dimensional coordinate data of each motion capture key point) as input data. The input dimension of the input data is 3×M×T, where T represents the number of consecutive motion frames, M represents the number of motion capture markers in each frame, and 3 represents the three-dimensional coordinates of the motion capture markers, i.e., the position information of the motion capture markers in the preset three-dimensional coordinates (x, y, z).

[0098] The feature embedding module mainly consists of several cascaded 1×1 convolutional layers. That is, it uses 1×1 convolution operations to change the number of channels in the input features, thereby embedding the input data into a specific high-dimensional feature space. Finally, the feature embedding module outputs embedded feature data with dimensions C×M×T, where C represents the number of channels in the embedded feature data.

[0099] In convolutional neural networks, a channel, also called a feature map, is used to identify the representational dimension of the embedded features. For example, in computer vision, a common channel refers to the RGB channels in an electronic image. A 64×64 RGB image can be represented by a 64×64×3 tensor. Here, 3 refers to the number of channels: red, green, and blue. In other words, all image features in the electronic image are represented by these three channels to varying degrees.

[0100] Furthermore, after obtaining the embedded feature data with the input dimension of C×M×T, the embedded features are input into the denoising network to construct the intra-frame relationship and the inter-frame relationship of the action through the intra-frame nonlocal module and the inter-frame nonlocal module of the denoising network.

[0101] For details, please refer to Figure 5 This is a schematic diagram of the structure of an intra-frame non-local module provided in another embodiment of this application.

[0102] The intra-frame non-local module includes: a first input module 501, a first compression network 502, a second compression network 503, a first fusion network 504, and a second fusion network 505.

[0103] In practical applications, the first input module 501 is used to receive the embedded feature data of all motion capture markers in a certain action frame. Since the intra-frame non-local module mainly considers the pairwise correlation between all motion capture markers in each action frame, the input dimension of the input data of the input module 501 here is M×C, where M represents the number of motion capture markers in each action frame, and C represents the number of channels of the embedded feature data. For ease of description of the embodiments of this application, F is used as the reference. in1 Embedded feature data that identifies all motion capture markers in a given action frame.

[0104] After obtaining the embedded feature data F in Then, the embedded feature data F in1 The first compression network 502 and the second compression network 503 are respectively input. The first compression network 502 and the second compression network 503 each include a 1×1 convolution (θ and θ). ) to the embedded feature data F in1 Channel compression is performed, and two first intermediate features θ(F) with dimensions of M×C / 2 are obtained respectively. in ) and second intermediate features Then, for the first intermediate feature θ(F) in1 ) and the second intermediate feature One of the intermediate features is transposed, and the transposed intermediate feature is multiplied by the other intermediate feature to obtain an M×M feature vector (in this embodiment, the above M×M feature vector is referred to as the first feature data) to initially characterize the correlation between different motion capture markers.

[0105] For example, suppose an action frame includes three different motion capture markers, namely motion capture marker 1, motion capture marker 2 and motion capture marker 3, and has a total of 4 channels (q, w, e, r).

[0106] Specifically, the first intermediate feature θ(F) in1 This can be represented by matrix 1:

[0107] Second intermediate feature It can be represented by matrix 2

[0108] After obtaining matrices 1 and 2, they are input into the first fusion network 504. The first fusion network 504 is used to convert the transposed first intermediate feature θ(F) into a single matrix. in1 ) and the second intermediate feature After matrix multiplication, the first feature data can be represented in the form of matrix 3:

[0109]

[0110] That is, the correlation between different motion capture markers is determined by multiplying the intermediate features by transposing them. However, it is important to consider that the correlation between these different motion capture markers is limited and affected by the channel type. Therefore, while considering the correlation between different motion capture markers, it is also necessary to consider the influence of the channel on different motion capture markers.

[0111] For example, the element (1q×2e+1w×2r) in the first row and second column of matrix 3 represents the relationship between motion capture marker 1 and motion capture marker 2 (including the relationship between motion capture marker 1 represented by channel q and motion capture marker 2 represented by channel e, and the relationship between motion capture marker 1 represented by channel w and motion capture marker 2 represented by channel r); at the same time, the element (2q×1e+2w×1r) in the second row and first column of matrix 3 also represents the relationship between motion capture marker 1 and motion capture marker 2 (including the relationship between motion capture marker 2 represented by channel q and motion capture marker 1 represented by channel e, and the relationship between motion capture marker 2 represented by channel w and motion capture marker 1 represented by channel r).

[0112] In one optional embodiment of this application, in order to weaken the influence of the above channels on the correlation between different motion capture markers, the embodiment of this application uses the Softmax function to process the above matrix.

[0113] To facilitate understanding of the role of the Softmax function in this embodiment, the max operation is explained below. Assume there are two values, a and b, where a > b. Taking max means taking a. However, the solution provided in this application requires comprehensive consideration of the feature data of motion capture markers under different channel representations. Therefore, we cannot remove any element from matrix 3 simply because the elements in the matrix are of different sizes. Thus, the Softmax function is used to process matrix 3, so that even if a > b, b will occasionally be taken, and the probability of taking b is related to its size.

[0114] After determining matrix 3 above, it is also necessary to restore the representation channels of each motion capture marker point through the second fusion network 505. Specifically, the restoration process is to convert F... in1 Multiplying this by matrix 3 yields an output matrix of dimension M×C, which in this application is specifically F. out1 ,Right now

[0115] Furthermore, after obtaining the aforementioned feature data F containing intra-frame relationships... out1 Afterwards, F out1 Input an inter-frame nonlocal module such that it includes the inter-frame relationship while also including the intra-frame relationship.

[0116] Please refer to Figure 6 This is a schematic diagram of the structure of an inter-frame non-local module provided in another embodiment of this application.

[0117] Similar to the intra-frame nonlocal module described above, the inter-frame nonlocal module includes a second input module 601, a third compression network 602, a fourth compression network 603, a third fusion network 604, and a fourth fusion network 605.

[0118] The input data of the second input module 601 is the F output of the intra-frame non-local module. out1 To facilitate the distinction between the inputs of the intra-frame nonlocal module and the inter-frame nonlocal module, and also to facilitate the description of the input data of the inter-frame nonlocal module, F will be used as the reference hereafter. in2 This represents the input data of the inter-frame nonlocal module. Further, since the input data F of the inter-frame nonlocal module... in2 The main consideration is the correlation between the same motion capture markers in different action frames, and F in2 The input data F already includes the relationships between different motion capture markers. in The dimension can be T×C, where T is the number of motion frames and C is the number of channels of the motion capture markers.

[0119] Similar to the intra-frame nonlocal modules mentioned above, in obtaining input data F in2 Then, first input data F in The data are input into the third compression network 602 and the fourth compression network 603, respectively. The third compression network 602 and the fourth compression network 604 each include 1×1 convolutions (β and ω) to process the embedded feature data F. in2 Channel compression is performed, and two third intermediate features β(F) with dimensions T×C / 2 are obtained respectively. in ) and the fourth intermediate feature ω(F in2 Then, for the third intermediate feature β(F) in ) and the fourth intermediate feature ω(F) in One of the intermediate features is transposed, and the transposed intermediate feature is multiplied by the other intermediate feature to obtain a T×T feature vector (in this embodiment, the above T×T feature vector is called the second feature data) to initially characterize the correlation between the same motion capture markers in different action frames.

[0120] For example, suppose we consider the correlation between motion capture markers 1 in different action frames, and the consecutive action frames include action frame 1, action frame 2 and action frame 3, with a total of 4 channels (a, s, d, f).

[0121] Specifically, the third intermediate feature β(F) in2 This can be represented by matrix 4.

[0122] Fourth intermediate feature ω(F) in2This can be represented by matrix 5.

[0123] After obtaining matrices 4 and 5, they are input into the third fusion network 604. The third fusion network 604 is used to convert the transposed third intermediate feature β(F) into a single matrix. in2 ) and the fourth intermediate feature ω(F in2 Multiplication of matrices yields the predicted feature data in matrix 6:

[0124]

[0125] That is, the association between the same motion capture markers in different action frames is determined by multiplying the intermediate features by transposing them. However, it should be considered that the association between the same motion capture markers in different action frames is still limited and affected by the channel type. Therefore, while considering the association between the same motion capture markers in different action frames, it is also necessary to consider the influence of the channel on the motion capture markers.

[0126] Similar to the intra-frame nonlocal module described above, in an optional embodiment of this application, the inter-frame nonlocal module also employs the Softmax function to weaken the influence of the aforementioned channel on the same motion capture marker points between different action frames.

[0127] After obtaining the aforementioned matrix 6, it is also necessary to restore the representation channels of each motion capture marker point through the fourth fusion network 605. Specifically, the restoration process involves converting F... in Multiplying by matrix 6 above yields an output matrix of dimension T×C. In this application, the output matrix is ​​specifically F. out2 That is, F out2 =Softmax(β(F in )*ω(F in2 ))*F in2 .

[0128] Among them, the F output by the inter-frame nonlocal module mentioned above in This refers to feature data that includes both inter-frame and intra-frame relationships within the action frame. This data considers both the spatial relationships between different motion capture markers in each action frame and the temporal relationships between the same markers in different action frames.

[0129] It should be noted that the above method of mining the inter-frame and intra-frame relationships of actions using an intra-frame non-local module and an inter-frame non-local module is only one optional implementation of this application and is not intended to limit the scope of protection of this application. In other implementations, the processing order of the data by the intra-frame non-local module and the inter-frame non-local module may also be that the inter-frame relationship of actions is mined first by the inter-frame non-local module, and then the intra-frame relationship of actions is mined by the intra-frame non-local module.

[0130] Furthermore, in an optional embodiment of this application, the denoising network may also include multiple interleaved intra-frame nonlocal modules and inter-frame nonlocal modules to continuously mine the inter-frame relationships and intra-frame relationships of the actions.

[0131] Furthermore, in order to predict the motion sequence of each motion capture keyframe of the target object in the action frame, the denoising network also includes several cascaded fully connected layers.

[0132] The fully connected layer is used to enable F out2 Information integration and dimensional transformation of the predicted feature data (i.e., the predicted feature data processed by the Softmax function) are used to predict the motion sequence of each motion capture marker point of the target object in the continuous action frames. Specifically, the dimension of the motion sequence can be represented as T×J×(3×3), where T represents the number of continuous action frames, J represents the number of motion capture key points of the target object in each action frame, and (3×3) represents the motion matrix of each motion capture key point.

[0133] To facilitate understanding of the data processing method provided in the above embodiments of this application, the data processing method will be described in detail below in conjunction with specific application scenarios.

[0134] Please refer to Figure 7 This is a schematic diagram illustrating an application scenario of a data processing method provided in another embodiment of this application.

[0135] like Figure 7 As shown, Figure 7 It includes: continuous motion frames 701, motion prediction model 702, and continuous motion sequence presentation module 703.

[0136] In this embodiment, the continuous motion frame refers to a series of video frames captured when an actor performs a certain action during the filming of a live-action computer animation (Computer Graphics, CG) movie. When the actor performs the action, key parts of his body are marked as motion capture markers, and the continuous video frames include the three-dimensional coordinate data of the motion capture markers.

[0137] After obtaining the three-dimensional coordinate data of each motion capture marker point in the continuous motion frame, the three-dimensional coordinate data is input into the motion prediction model 702.

[0138] The motion prediction model 702 includes: a feature embedding module 7021 and a denoising network 7022.

[0139] The feature embedding module 7021 is used to convert the three-dimensional coordinate data into input feature data suitable for input into the denoising network.

[0140] The denoising network 7022 includes several intra-frame nonlocal modules 7022-1 and inter-frame nonlocal modules 7022-2 that are superimposed in an interleaved manner. The denoising network 7022 also includes a fully connected layer 7022-3.

[0141] Among them, the intra-frame non-local module 7022-1 is used to mine the spatial correlation of different motion capture markers in each action frame, and the inter-frame non-local module 7022-2 is used to mine the temporal correlation of the same markers in each action frame.

[0142] After processing by the intra-frame nonlocal module 7022-1 and the inter-frame nonlocal module 7022-2, the feature data carrying the spatial correlation of the different motion capture markers and the temporal correlation of the same motion capture markers in each action frame is input into the fully connected layer 7022-3. The fully connected layer 7022-3 integrates information and changes the dimensions of the input data to obtain the motion sequence of each motion capture marker in the continuous action frames, and inputs the motion sequence into the action sequence presentation module 703.

[0143] Specifically, in practical applications, a preset number of consecutive action frames can be selected as sample action frames. Then, the second feature data of the sample action frames is extracted using the aforementioned inter-frame non-local module and intra-frame non-local module. The second feature data of the sample action frames and the actual action sequence of the motion capture markers in the consecutive action frames are used to train a convolutional neural network to obtain the data.

[0144] The motion sequence rendering module 703 is used to render the motion sequence of each motion capture marker point output by the fully connected layer, and restore the motion process of the actor in the continuous video frames.

[0145] In summary, the data processing method provided in this application, after obtaining the initial feature data of motion capture markers of the target object in continuous action frames, mines the correlation between different motion capture markers in each action frame and the correlation between the same motion capture markers in different action frames based on a motion prediction model. Then, combining the correlation between different motion capture markers in adjacent action frames and the correlation between the same motion capture markers in different action frames, it predicts the motion sequence of the target object's motion capture markers in continuous action frames. This method not only considers the spatial relationship between different motion capture markers in each action frame but also the temporal correlation of motion capture markers in different action frames, thus completing the prediction of the target object's motion process. Simultaneously, it achieves denoising processing of continuous action frames. This method reduces the denoising cost of motion capture data and improves the denoising efficiency of motion capture data.

[0146] Another embodiment of this application provides a training method for a motion prediction model; please refer to [reference needed]. Figure 8 This is a flowchart of a training method for a motion sequence prediction model provided in another embodiment of this application.

[0147] like Figure 8 As shown, the training method of the motion prediction model includes the following steps S801 to S805.

[0148] Step S801: Obtain continuous motion frames of at least one captured sample object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the sample object.

[0149] Specifically, the continuous motion frames of the sample object can be understood as continuous motion frames to be denoised obtained through the Internet. However, it should be noted that since this embodiment is mainly used to optimize and train the motion sequence prediction model, while obtaining the continuous motion frames of the sample object, it is also necessary to ensure that the actual motion sequence of each motion capture marker point of the sample object in the continuous motion frames is known, that is, the noiseless motion sequence of the sample object in the continuous motion frames is known.

[0150] In practical applications, the continuous action frames of the sample object can be obtained from the Internet, or they can be retained during the work of relevant personnel. This application does not impose any restrictions on this.

[0151] Specifically, step S801 is basically the same as step S101 in the previous embodiment, except that the object captured in this embodiment is a sample object, while the object captured in step S101 is the target object. For relevant details, please refer to the above description of step S101; it will not be repeated here.

[0152] Step S802: Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames;

[0153] Specifically, step S802 is basically the same as step S102 in the previous embodiment. For relevant details, please refer to the description of step S101 above. It will not be repeated here.

[0154] Step S803: Input the initial feature data into the motion prediction model, obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model, and obtain the first action sequence of at least one motion capture marker of the sample object in the continuous action frames based on the predicted feature data; wherein, the intra-frame relationship includes the association between different markers in each action frame, and the inter-frame relationship includes the association between the same markers in adjacent action frames.

[0155] Specifically, step S803 is basically the same as step S103 in the previous embodiment, except that the denoising and prediction object of the denoising network is changed from the target object to the sample object. For relevant details, please refer to the description of step S103 above; it will not be repeated here.

[0156] Step S804: Obtain the actual motion sequence of the motion capture marker of the sample object in the continuous motion frames.

[0157] In this embodiment of the application, the actual action sequence of the motion capture marker is the denoised action sequence. The actual action sequence is used to compare with the first action sequence obtained by the motion prediction model to verify the accuracy of the motion prediction model.

[0158] Step S805: Optimize the parameters of the motion prediction model based on the first motion sequence and the actual motion sequence to obtain a parameter-optimized motion prediction model.

[0159] In practical applications, the above step S805 mainly includes:

[0160] A loss function is constructed based on the difference between the first motion sequence obtained by the motion prediction model and the actual motion sequence, and the parameters of the motion prediction model are optimized based on the loss function.

[0161] Specifically, in this embodiment, the loss function is the sum of the absolute values ​​of the differences between the first motion sequence and the actual motion sequence. The smaller the value of the loss function, the more accurate the prediction result of the motion prediction model for the motion sequence.

[0162] Furthermore, optimizing the parameters of the motion prediction model refers to iteratively updating the loss function through gradient descent, thereby continuously optimizing the weight parameters of the motion prediction model so that the first motion sequence output by the motion prediction model is as consistent as possible with the actual motion sequence, thus obtaining the desired complete model.

[0163] In one optional embodiment of this application, the optimization of the motion model is typically achieved through multiple iterations using initial feature data from different sample objects. That is, after completing one parameter optimization of the motion prediction model, it is further determined whether the number of iterations in the optimization process meets the preset number of iterations. If the determination result is not, other sample objects are further selected and the execution of step S801 is returned to complete the iterative optimization.

[0164] Another embodiment of this application provides a data processing apparatus. This apparatus embodiment is basically similar to the data processing method embodiment provided in this application, so the description is relatively simple. For relevant parts, please refer to the above description of the data processing method embodiment, and it will not be repeated here.

[0165] Please refer to Figure 9 This is a schematic diagram of the structure of a data processing device provided in another embodiment of this application.

[0166] The device includes:

[0167] The first acquisition unit 901 is used to acquire at least one captured continuous motion frames of a target object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the target object.

[0168] The second acquisition unit 902 acquires the initial feature data of the motion capture marker in the continuous action frames based on the continuous action frames.

[0169] The model prediction unit 903 is used to input the initial feature data into the motion prediction model, and obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the predicted feature data; wherein, the intra-frame relationship includes the association between different markers in each motion frame, and the inter-frame relationship includes the association between the same markers in each motion frame.

[0170] Optionally, obtaining the initial feature data of the motion capture marker in the continuous motion frames based on the continuous motion frames includes:

[0171] The initial feature data of the target object in the continuous action frames are obtained based on the number of consecutive action frames, the number of motion capture markers in each action frame, and the three-dimensional coordinates of each motion capture marker.

[0172] Optionally, the denoising network includes an inter-frame nonlocal module and an intra-frame nonlocal module, wherein the inter-frame nonlocal module is used to determine the inter-frame relationship of the action, and the intra-frame nonlocal module is used to determine the intra-frame relationship of the action.

[0173] Optionally, the step of inputting the initial feature data into the motion prediction model and obtaining predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model includes:

[0174] The initial feature data is input into the motion prediction model, and the initial feature data is processed by the intra-frame nonlocal module to obtain first feature data that has the correlation between different motion capture markers in each action frame.

[0175] The first feature data is processed by the inter-frame non-local module, and the predicted feature data is obtained based on the processing result. The predicted feature data includes the correlation between different motion capture markers in each action frame and the correlation between the same motion capture markers in adjacent action frames.

[0176] Optionally, the step of processing the initial feature data through the intra-frame non-local module to obtain first feature data containing the correlation between different motion capture marker points in each action frame includes:

[0177] The initial feature data is transposed to obtain the transposed embedded feature data.

[0178] The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels.

[0179] Multiply the transposed first or second intermediate feature with another intermediate feature to obtain the first feature data that has the correlation relationship between different motion capture marker points in the action frame.

[0180] Optionally, the step of processing the first feature data through the inter-frame non-local module and obtaining the predicted feature data based on the processing result includes:

[0181] The first feature data is compressed by the third and fourth compression networks of the inter-frame nonlocal module to obtain the third and fourth intermediate features that represent the image features of the continuous action frames using different channels.

[0182] The transposed third or fourth intermediate feature is multiplied by another intermediate feature to obtain the predicted feature data.

[0183] Optionally, the step of inputting the initial feature data into the motion prediction model and obtaining predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model includes:

[0184] The initial feature data is input into the motion prediction model, and the initial feature data is processed by the inter-frame nonlocal module to obtain second feature data that has the correlation between the same motion capture marker points in adjacent action frames.

[0185] The second feature data is processed by the intra-frame non-local module, and the predicted feature data is obtained based on the processing result. The predicted feature data includes the correlation between identical motion capture markers in adjacent action frames and the correlation between different motion capture markers in each action frame. Optionally, the denoising network further includes a fully connected layer, which is used to predict the action sequence of each marker of the target object in the consecutive action frames based on the predicted feature data.

[0186] Another embodiment of this application provides a training device for a motion prediction model. This device embodiment is basically similar to the training method of the motion prediction model provided in this application, so the description is relatively simple. For relevant parts, please refer to the introduction of the training method of the motion prediction model, which will not be repeated here.

[0187] Please refer to Figure 10 This is a schematic diagram of the training device structure for a motion prediction model provided in another embodiment of this application.

[0188] The device includes:

[0189] The third acquisition unit 1001 is used to acquire continuous motion frames of at least one captured sample object, wherein each motion frame of the continuous motion frame is marked with two or more motion capture markers of the sample object.

[0190] The fourth acquisition unit 1002 is used to acquire the initial feature data of the motion capture marker in the continuous action frames based on the continuous action frames;

[0191] The sample prediction unit 1003 is used to input the initial feature data into the motion prediction model, obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains a first action sequence of at least one motion capture marker of the sample object in the consecutive action frames based on the predicted feature data; wherein, the intra-frame relationship includes the association relationship between different markers in each action frame, and the inter-frame relationship includes the association relationship between the same markers in adjacent action frames;

[0192] The fifth acquisition unit 1004 is used to obtain the actual motion sequence of the motion capture marker points of the sample object in the continuous motion frames;

[0193] The model training unit 1005 is used to optimize the model parameters of the motion prediction model based on the first motion sequence and the actual motion sequence, so as to obtain a motion prediction model with optimized parameters.

[0194] Optionally, optimizing the parameters of the motion prediction model based on the first motion sequence and the actual motion sequence includes:

[0195] A loss function is constructed based on the difference between the first motion prediction sequence and the actual motion sequence, and the motion model is optimized based on the loss function.

[0196] Another embodiment of this application provides an electronic device, please refer to... Figure 11 This is a schematic diagram of an electronic device structure provided in another embodiment of this application.

[0197] The electronic device includes: a processor 1101;

[0198] The memory 1102 is used to store the program of the method, which is executed by the processor 1101 when it is read and executed, performing any of the methods in the above embodiments.

[0199] Another embodiment of this application provides a computer storage medium storing a computer program that, when executed, performs any of the methods described in the above embodiments.

[0200] It should be noted that the detailed description of the electronic device and computer storage medium provided in the embodiments of this application can be found in the relevant description of the above method embodiments provided in this application, and will not be repeated here.

[0201] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this invention. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

[0202] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0203] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0204] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0205] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A data processing method, characterized in that, include: Acquire continuous motion frames of at least one captured target object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the target object; Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames; The initial feature data is input into the motion prediction model, and through the denoising network of the motion prediction model, predicted feature data containing inter-frame and intra-frame relationships of actions is obtained; and The motion prediction model obtains the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the prediction feature data; The intra-frame relationship of the action includes the association between different motion capture markers in each action frame, and the inter-frame relationship of the action includes the association between the same motion capture markers in adjacent action frames. The denoising network includes an intra-frame nonlocal module, which is used to determine the intra-frame relationship of the action; The step of obtaining predictive feature data with inter-frame and intra-frame relationships of actions through the denoising network of the motion prediction model includes: The initial feature data is transposed to obtain the transposed embedded feature data. The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels. Multiply the transposed first or second intermediate feature with another intermediate feature to obtain first feature data that contains the correlation between different motion capture markers in each action frame.

2. The method according to claim 1, characterized in that, The step of obtaining the initial feature data of the motion capture marker points in the continuous motion frames includes: The initial feature data of the target object in the continuous action frames are obtained based on the number of consecutive action frames, the number of motion capture markers in each action frame, and the three-dimensional coordinates of each motion capture marker.

3. The method according to claim 1, characterized in that, The denoising network includes an inter-frame nonlocal module, wherein the inter-frame nonlocal module is used to determine the inter-frame relationship of the action.

4. The method according to claim 3, characterized in that, The step of obtaining predictive feature data with inter-frame and intra-frame relationships of actions through the denoising network of the motion prediction model includes: The first feature data is processed by the inter-frame non-local module, and the predicted feature data is obtained based on the processing result. The predicted feature data includes the correlation between different motion capture markers in each action frame and the correlation between the same motion capture markers in adjacent action frames.

5. The method according to claim 4, characterized in that, The step of processing the first feature data through the inter-frame non-local module and obtaining the predicted feature data based on the processing result includes: The first feature data is compressed by the third and fourth compression networks of the inter-frame nonlocal module to obtain the third and fourth intermediate features that represent the image features of the continuous action frames using different channels. The transposed third or fourth intermediate feature is multiplied by another intermediate feature to obtain the predicted feature data.

6. The method according to any one of claims 4-5, characterized in that, The denoising network further includes a fully connected layer, which is used to predict the action sequence of each marker point of the target object in the continuous action frames based on the prediction feature data.

7. A training method for a motion prediction model, characterized in that, include: Acquire continuous motion frames of at least one captured sample object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the sample object; Based on the continuous action frames, obtain the initial feature data of the motion capture marker points in the continuous action frames; The initial feature data is input into the motion prediction model, and the denoising network of the motion prediction model is used to obtain predicted feature data containing inter-frame and intra-frame relationships of actions; and The motion prediction model obtains a first motion sequence of at least one motion capture marker of the sample object in the consecutive motion frames based on the prediction feature data; wherein, the intra-motion frame relationship includes the association relationship between different markers in each motion frame, and the inter-motion frame relationship includes the association relationship between the same markers in adjacent motion frames; Obtain the actual motion sequence of the motion capture markers of the sample object in the continuous motion frames; Based on the first action sequence and the actual action sequence, the model parameters of the motion prediction model are optimized to obtain a parameter-optimized motion prediction model. The denoising network includes an intra-frame nonlocal module, which is used to determine the intra-frame relationship of the action; The step of obtaining predictive feature data with inter-frame and intra-frame relationships of actions through the denoising network of the motion prediction model includes: The initial feature data is transposed to obtain the transposed embedded feature data. The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels. Multiply the transposed first or second intermediate feature with another intermediate feature to obtain first feature data that contains the correlation between different motion capture markers in each action frame.

8. The method according to claim 7, characterized in that, The step of optimizing the model parameters of the motion prediction model based on the first action sequence and the actual action sequence includes: A loss function is constructed based on the difference between the first action sequence and the actual action sequence, and the motion model is optimized based on the loss function.

9. A data processing apparatus, characterized in that, include: The first acquisition unit is used to acquire at least one continuous motion frame of a target object, wherein each motion frame of the continuous motion frame is marked with two or more motion capture markers of the target object. The second acquisition unit is used to acquire the initial feature data of the motion capture marker in the continuous action frames based on the continuous action frames. The model prediction unit is used to input the initial feature data into the motion prediction model, and obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains the motion sequence of at least one motion capture marker of the target object in the continuous motion frames based on the predicted feature data. The intra-frame relationship of the action includes the association between different motion capture markers in each action frame, and the inter-frame relationship of the action includes the association between the same motion capture markers in adjacent action frames. The denoising network includes an intra-frame nonlocal module, which is used to determine the intra-frame relationship of the action; The step of obtaining predictive feature data with inter-frame and intra-frame relationships of actions through the denoising network of the motion prediction model includes: The initial feature data is transposed to obtain the transposed embedded feature data. The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels. Multiply the transposed first or second intermediate feature with another intermediate feature to obtain first feature data that contains the correlation between different motion capture markers in each action frame.

10. A training device for a motion prediction model, characterized in that, include: The third acquisition unit is used to acquire at least one captured continuous motion frames of a sample object, wherein each motion frame of the continuous motion frames is marked with two or more motion capture markers of the sample object. The fourth acquisition unit is used to acquire the initial feature data of the motion capture marker in the continuous motion frames based on the continuous motion frames. A sample prediction unit is configured to input the initial feature data into a motion prediction model, obtain predicted feature data with inter-frame and intra-frame relationships through the denoising network of the motion prediction model; and the motion prediction model obtains a first action sequence of at least one motion capture marker of the sample object in the consecutive motion frames based on the predicted feature data; wherein the intra-frame relationship includes the association between different markers in each motion frame, and the inter-frame relationship includes the association between the same markers in adjacent motion frames; The fifth acquisition unit is used to obtain the actual motion sequence of the motion capture marker points of the sample object in the continuous motion frames; The model training unit is used to optimize the model parameters of the motion prediction model based on the first action sequence and the actual action sequence, so as to obtain the parameter-optimized motion prediction model. The denoising network includes an intra-frame nonlocal module, which is used to determine the intra-frame relationship of the action; The step of obtaining predictive feature data with inter-frame and intra-frame relationships of actions through the denoising network of the motion prediction model includes: The initial feature data is transposed to obtain the transposed embedded feature data. The embedded feature data is compressed by the first compression network and the second compression network of the intra-frame nonlocal module to obtain the first intermediate feature and the second intermediate feature that represent the image features of the continuous action frames using different channels. Multiply the transposed first or second intermediate feature with another intermediate feature to obtain first feature data that contains the correlation between different motion capture markers in each action frame.

11. An electronic device, characterized in that, include: processor; A memory for storing a program of a method, which, when read and executed by a processor, performs the method according to any one of claims 1-8.

12. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when read and executed by a processor, performs the method described in any one of claims 1-8.

Images