Three-dimensional motion sequence generation method, device and storage medium

Abstract

Embodiments of the present application provide a three-dimensional action sequence generation method and device and a storage medium, which belong to but are not limited to the technical field of artificial intelligence. The method comprises: performing time sequence feature coding on a three-dimensional action input sequence and target condition feature matrix data through a conditional diffusion model to obtain an action time sequence feature sequence; sampling the action time sequence feature sequence using a sliding window with a fixed window length according to a preset sliding step to obtain window features corresponding to the sliding window one by one; performing feature extraction and fusion processing on the multiple window features according to a time axis, and performing action sequence prediction on the fused continuous action features to obtain a three-dimensional action sequence of the current round of reasoning, and in the case that the conditional diffusion model meets a reasoning stop condition, taking the three-dimensional action sequence as a target three-dimensional action sequence. The target three-dimensional action sequence generated by the embodiments of the present application can restore local details while taking into account the continuity of local details.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device and storage medium for generating three-dimensional action sequences. Background Technology

[0002] With the development of technologies such as digital humans, virtual reality, augmented reality, interactive teaching, and digital cultural asset management, the reconstruction and automated generation of 3D movements in traditional dance / opera and other physical performances has become an important direction for cultural preservation and content production. Existing methods mostly focus on everyday movements, sports movements, or general dance movements, primarily aiming for global semantic consistency or beat synchronization in generating 3D movement sequences. However, for traditional physical performances with strong stylistic and symbolic elements, relying solely on global context modeling makes it difficult to stably recreate local details while maintaining their continuity. Therefore, there is an urgent need for a method to generate 3D movement sequences that can recreate local details while ensuring their continuity. Summary of the Invention

[0003] The main objective of this application is to propose a method, device, and storage medium for generating three-dimensional action sequences, which can restore local details while maintaining the continuity of those details.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for generating three-dimensional action sequences, the method comprising: Multimodal feature extraction is performed on the multimodal data to obtain initial conditional feature matrix data, and target conditional feature matrix data is determined based on the initial conditional feature matrix data; the target conditional feature matrix data includes skeletal prior features and multimedia conditional constraint features; Based on the target condition feature matrix data, the target three-dimensional action sequence is obtained by performing three-dimensional action sequence reasoning on the three-dimensional action input sequence through the condition diffusion model. The three-dimensional action sequence reasoning is used to perform at least one round of action reasoning, and in each round of action reasoning, the conditional diffusion model performs the following steps: Temporal feature encoding is performed on the three-dimensional action input sequence and the target condition feature matrix data to obtain the action temporal feature sequence; According to a preset sliding step size, a sliding window of fixed length is used to sample the action timing feature sequence to obtain window features that correspond one-to-one with the sliding window; the length of the sliding step size is less than the length of the window; each sliding window is used to sample a single action unit; The multiple window features are extracted and fused according to the time axis, and the continuous action features of the fused process are used to predict the action sequence to obtain the three-dimensional action sequence of the current round of reasoning.

[0005] To achieve the above objectives, a second aspect of this application provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When at least one of the programs is executed by at least one of the processors, the three-dimensional action sequence generation method as described in any one of the first aspects is implemented.

[0006] To achieve the above objectives, a third aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which are configured to be executed by a processor as described in any one of the first aspects of the three-dimensional action sequence generation method.

[0007] The 3D action sequence generation method, device, and storage medium proposed in this application, after obtaining the target conditional feature matrix data, first encodes the default 3D action sequence and the target conditional feature matrix data using a conditional diffusion model to form a time-varying action temporal feature sequence constrained by the target conditional feature data. Then, by setting a sliding window with a fixed sliding step size for sampling, window features recording the features of each action unit can be obtained. Since the sliding step size is less than the window length, the features sampled by adjacent sliding windows partially overlap. At this point, feature extraction and fusion processing of the window features along the time axis can achieve the fusion of features in the overlapping area. This allows the same boundary action to be simultaneously constrained by two preceding and following local contexts, thus maintaining the continuity of posture, velocity, and acceleration, and reducing the probability of boundary jumps caused by hard segmentation. Therefore, compared with related technologies, the target 3D action sequence generated by the embodiments of this application can restore local details while maintaining the continuity of local details. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating an embodiment of the three-dimensional action sequence generation method provided in this application; Figure 2 This is a schematic diagram of the network architecture of the conditional diffusion model in one embodiment of the three-dimensional action sequence generation method provided in this application; Figure 3 This is a schematic diagram of the network structure of the local self-attention module in the conditional diffusion model of one embodiment of the three-dimensional action sequence generation method provided in this application; Figure 4 This is a schematic diagram of the network architecture of the cross-modal semantic alignment network in one embodiment of the three-dimensional action sequence generation method provided in this application; Figure 5This is a schematic diagram of the data processing flow of an embodiment of the application of the target three-dimensional action sequence in the three-dimensional action sequence generation method provided in this application; Figure 6 This is a schematic diagram of data flow in one embodiment of the three-dimensional motion sequence generation method provided in this application applied to the generation of three-dimensional motion sequences for Dunhuang dance; Figure 7 This is a schematic diagram of the hardware structure corresponding to the three-dimensional action sequence generation method provided in this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0010] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] With the development of digital humans, virtual reality, augmented reality, interactive teaching, and digital cultural asset management technologies, the reconstruction and automated generation of 3D movements in traditional dance / opera and other physical performances has become an important direction for cultural preservation and content production. Existing methods mostly focus on everyday movements, sports movements, or general dance movements, primarily aiming at global semantic consistency or rhythmic synchronization to generate 3D movement sequences. However, for traditional physical performances with strong stylistic and symbolic elements, relying solely on global context modeling makes it difficult to stably reconstruct local details while maintaining their continuity. For example, consider dance movements such as "playing the pipa backwards," "turning and supporting from the side," "smelling flowers," and "both hands supporting with slight bends." These movements involve not only overall posture but also fine-grained forms such as wrist bends, fingertip extensions, shoulder and hip twists, torso tilts, and weight shifts. If the model only focuses on global sequence correlation, it establishes connections between each frame and all other frames, easily overemphasizing long-term averaged trends, thus weakening details such as wrist bends, fingertip directions, hip twists, and shoulder and neck curves that appear within a short timeframe. Furthermore, long-term sequence generation involves joint jitter and boundary jumps, resulting in a large computational load. If, to reduce computation, local windows are used for sequential sampling, the lack of information exchange between independent local windows can easily lead to speed discontinuities, abrupt pose changes, and motion stuttering at cross-window points. Therefore, existing methods for generating 3D motion sequences are more prone to causing poor motion transitions, speed abrupt changes, or semantic breaks when applied to strong-style and strongly symbolic sequences. Thus, there is an urgent need for a 3D motion sequence generation method that can restore local details while maintaining their continuity. Based on this, embodiments of this application provide a 3D motion sequence generation method, device, and storage medium that can restore local details while maintaining their continuity.

[0013] Understandably, referring to Figure 1 As shown, a three-dimensional action sequence generation method according to an embodiment of this application includes: Step S100: Perform multimodal feature extraction on the multimodal data to obtain initial conditional feature matrix data, and determine the target conditional feature matrix data based on the initial conditional feature matrix data; the target conditional feature matrix data includes skeletal prior features and multimedia conditional constraint features; Step S200: Based on the target condition feature matrix data, perform three-dimensional action sequence reasoning on the three-dimensional action input sequence through the condition diffusion model to obtain the target three-dimensional action sequence; Specifically, the three-dimensional action sequence reasoning is used to perform at least one round of action reasoning, and in each round of action reasoning, the conditional diffusion model performs the following steps: Temporal feature encoding is performed on the 3D action input sequence and the target condition feature matrix data to obtain the action temporal feature sequence; Based on a preset sliding step size, a sliding window of fixed length is used to sample the action timing feature sequence to obtain window features that correspond one-to-one with the sliding window; the length of the sliding step size is less than the length of the window; each sliding window is used to sample a single action unit. The system extracts and fuses features from multiple windows along a timeline, and predicts action sequences from the fused continuous action features to obtain the three-dimensional action sequence for the current round of reasoning.

[0014] After obtaining the target conditional feature matrix data, a conditional diffusion model is used to first encode the default 3D action sequence and the target conditional feature matrix data into temporal features, forming an action temporal feature sequence that changes over time and is constrained by the target conditional feature data. Then, by setting a sliding window with a fixed sliding step size for sampling, window features recording the features of each action unit can be obtained. Since the sliding step size is less than the window length, the features sampled by adjacent sliding windows partially overlap. At this point, feature extraction and fusion processing of the window features along the time axis can achieve the fusion of features in the overlapping area. This allows the same boundary action segment to be simultaneously constrained by two local contexts, thus maintaining the continuity of posture, velocity, and acceleration, and reducing the probability of boundary jumps caused by hard segmentation. Therefore, the target 3D action sequence generated in this embodiment can restore local details while maintaining the continuity of those details.

[0015] The method described in this application can be applied to Dunhuang dance, as well as to the generation of three-dimensional motion input sequences for other traditional dances or operas with strong style and strong local gesture symbols.

[0016] This application does not limit the type of conditional diffusion model. In actual operation, any one of the following can be selected: denoising network model, U-Net model, Transformer model, temporal convolutional network model, state space model, or other sequence modeling network. This application does not limit this.

[0017] This application does not limit the method of multimodal feature extraction for multimodal data. For example, a cross-modal feature encoder can be used for feature extraction. The cross-modal feature encoder may include at least one of a pre-trained language model, an audio self-supervised model, a visual-language model, or a specially trained dance motion encoder. This application does not limit the specific methods used in this regard.

[0018] The target 3D motion sequence can be represented in various ways, including but not limited to at least one of the following: root node displacement, root node orientation, local joint rotation, joint 3D coordinates, hand keypoints, foot contact state, and body center of gravity position. By using the above combined representation, bone length drift caused by using only joint coordinates can be reduced, and subsequent motion retargeting based on the target 3D motion sequence can be facilitated.

[0019] The conditional diffusion model in this application embodiment can be an independently deployed model or a composite model composed of different types of network models. This application embodiment does not limit this.

[0020] The 3D action input sequence for the first round of action reasoning is a pre-configured 3D action input sequence. The 3D action input sequence for each round of action reasoning after the first round is the 3D action sequence output from the previous round. The pre-configured 3D action input sequence can be randomly generated, and can be valid or invalid action sequences. In practical applications, this can be selectively set according to the chosen conditional diffusion model. For example, if the conditional diffusion model is a denoising network model, the pre-configured 3D action input sequence is an invalid action sequence, which is a randomly generated noise sequence. In the case of multiple rounds of action reasoning, this application embodiment does not restrict the stopping condition for 3D action sequence reasoning. In some embodiments, a fixed number of reasoning rounds can be set; in other embodiments, a credibility analysis can be performed on the output 3D action sequence, and reasoning can stop when a preset credibility is met. This application embodiment will not elaborate on these points further.

[0021] The target conditional feature matrix data is used to impose motion constraints on the reasoning process of three-dimensional action input sequences. It is also used to record the conditional constraint features of multiple modalities.

[0022] Each action unit corresponds to a recognizable action. This application does not limit how the sliding step size is set. In some embodiments, it can be set according to the density between two action units in a normal video 3D action sequence. For example, in some embodiments, if the window length of the sliding window is L, the sliding step size is set between 0.25 times the window length and 0.5 times the window length, so that the overlap area ratio of the overlapping region between adjacent windows is between 50% and 75%. For example, 0.25 times the window length or 0.5 times the window length can be selected. For example, if the motion sequence is sampled at 30fps or 60fps, the window length L of the sliding window can be selected as 16, 32, 48, or 64 frames, corresponding to a local motion segment of approximately 0.5 to 2 seconds. In this case, when L is 16 frames, the sliding step s is 8 frames; when L is 32 frames, s is 16 frames. This allows for the extraction of complete local motion semantics for dances like Dunhuang dance, which features slight wrist flexion, fingertip extension, shoulder and hip twisting, torso tilting, and weight shifting within a short timeframe. It also ensures that the computational load is not reduced due to excessively long sliding windows, thus avoiding weakening local details or increasing computational complexity. In other words, the window length L covers a relatively complete local motion unit, and the sliding step s controls the information transfer density between adjacent motion segments.

[0023] Feature extraction fusion processing involves fusing features of the overlapping region between two adjacent sliding windows and combining the features of the overlapping region with the features of the non-overlapping region that is not covered in each sliding window. This allows the same boundary action to be constrained by two local contexts simultaneously, thereby maintaining the continuity of posture, velocity and acceleration and reducing the probability of boundary jumps caused by hard segmentation.

[0024] This application does not limit the types of modalities included in the multimodal data. In some embodiments, the multimodal data may include multimodal data with a single modality or multimodal data with multiple modalities. The modality may be at least one of text, raw image (including at least one of images and videos), music rhythm, and action sequence. The text may be a text keyword description or a complete statement. For example, the application of the three-dimensional action sequence generation method of this application embodiment under different modal input conditions is described below: Example 1: Text-described Dunhuang dance motion generation: The user-input style-specific text keywords, "playing the pipa backwards, graceful posture, twisting hips, and agile hand gestures," are used as multimodal data for the text modality. Multimodal feature extraction is performed on these keywords to obtain text style-specific features. These text features include core primitive features and style / posture constraint features. The core primitive features identify "playing the pipa backwards" as the core action primitive, while the style / posture constraint features identify "graceful posture," "twisting hips," and "agile hand gestures" as style and local posture constraints. Initially, the feature condition matrix data only includes text features. Based on the text features, the corresponding pipa music theme, torso twisting rules, hand posture rules, and corresponding skeletal priors are retrieved from the knowledge base to obtain the conditional features corresponding to each modality of music rhythm, original image, and action sequence. The text features and each conditional feature are combined and aligned to obtain the target conditional matrix data. At this time, action inference is performed based on the conditional diffusion model and the target conditional matrix data. During the inference process, the wrist angle, shoulder and hip reverse twist, and fingertip direction change can be captured through overlapping sliding windows. Finally, a three-dimensional skeletal action sequence matching the text space description (that is, the above-mentioned target three-dimensional action sequence) is generated.

[0025] Example 2: Dynamic choreography driven by music rhythm: The audio accompaniment for a Dunhuang dance, input by the user, is used as multimodal data for the audio modality. Rhythmic conditional features are obtained by extracting features corresponding to beat points, velocity curves, energy envelopes, and musical phrase structures from this audio accompaniment. Based on a pre-defined knowledge base, conditional feature completion is performed on the rhythmic conditional features to obtain target conditional feature matrix data. Then, based on the constraints of the target conditional feature data, the conditional diffusion model can effectively adjust the speed and amplitude of movements according to the beat intensity during the generation of the target 3D motion sequence: generating more pronounced footwork, upper limb extensions, or trunk twists on strong beats or climaxes, and generating smoother gesture extensions and weight shifts on weak beats or transitional sections. At this point, through beat synchronization loss and foot contact constraints based on the target conditional feature matrix data, the generated target 3D motion sequence maintains a high degree of temporal consistency with the changes in the intensity of the accompaniment.

[0026] Example 3: Historical Video Reconstruction and Multi-role Redirection: The input historical dance teaching video is used as multimodal data. Two-dimensional key points are extracted from the historical dance teaching video to obtain skeletal prior feature data. Based on a preset knowledge base, the skeletal prior feature data is conditionally completed to obtain target conditional feature matrix data. At this time, the conditional diffusion model can reconstruct three-dimensional skeletal movements based on the target conditional feature data. At this time, the method based on the embodiment of this application can compensate for and repair the missing key points caused by overlapping clothing and limbs by using the temporal consistency of previous and subsequent frames, bone length constraints, and motion priors, and transfer the style to different virtual characters such as flying apsaras, bodhisattvas, or warriors through redirection.

[0027] Example 4: Generation of long sequences with joint control of text and audio: The user-inputted action text keywords and accompanying audio are used as multimodal data. Assuming the action text keywords specify "flying theme, arms outstretched, turning and looking back," and the accompanying audio provides a specific rhythm, the text keywords and accompanying audio are encoded separately (i.e., feature extraction), and their conditional weights are determined for each time period. For example, spatial constraint weights are increased when the text specifies a specific posture, and rhythmic constraint weights are increased when there is a downbeat in the accompanying audio. Then, the remaining conditional features, supplemented by a knowledge base, are aligned based on their conditional weights to obtain the target conditional feature matrix data. This target conditional feature matrix data and a conditional diffusion network are used for action inference. During action inference, overlapping windows are used to maintain natural connections between adjacent segments, reducing posture drift and rhythmic misalignment common in long sequences in the output target 3D action sequence.

[0028] Understandably, the conditional diffusion model includes a conditional injection module, which comprises an input embedding layer, a linear mapping layer, and a temporal feature extraction layer. Temporal feature encoding is performed on the 3D action input sequence and the target conditional feature matrix data to obtain the action temporal feature sequence, including: The 3D action input sequence is discretized and conditionally vectorized by the input embedding layer to obtain the dense feature vector of each frame within the time period corresponding to the 3D action input sequence. The linear mapping layer performs linear mapping transformation on each dense feature vector to obtain linear mapping features; The temporal location features are extracted from the linear mapping features through the temporal feature extraction layer to obtain the temporal location encoded features; Based on the temporal location coding features and linear mapping features, the basic temporal features are obtained; Based on the conditional features in the target conditional feature matrix data, the basic temporal features are modulated to obtain the action temporal feature sequence.

[0029] This application does not limit the modulation method. In some embodiments, FiLM modulation can be used based on the modulation parameters corresponding to all conditional features in the target conditional feature matrix data. In other embodiments, ADA modulation can be used based on the modulation parameters corresponding to all conditional features in the target conditional feature matrix data. This application will not elaborate on these specific methods. For example, in some embodiments, modulation parameters can be generated based on all conditional features in the target conditional feature matrix data. In other embodiments, taking the conditional diffusion model as a denoising network as an example, modulation parameters can be generated based on noise step encoding (i.e., noise features of the current round), temporal position encoding, linear mapping features, and the target conditional feature matrix data. This allows the basic temporal features to be modulated based on the modulation parameters when using FiLM, ADA, or other modulation methods to obtain an action temporal feature sequence.

[0030] The embodiments of this application do not limit the specific structure of the input embedding layer, the linear mapping layer, and the temporal feature extraction layer. Those skilled in the art can selectively set them according to actual needs.

[0031] The temporal feature extraction layer can be a temporal convolutional layer or a Transformer coding layer. In this case, the skeletal representation of each frame can be converted into a hidden coding layer of uniform dimension through the conditional injection module.

[0032] Understandably, the conditional diffusion model also includes a temporal residual module. Before obtaining the window features that correspond one-to-one with the sliding window, the method also includes: The action timing features are updated by performing residual processing on the timing features through the timing residual module.

[0033] Understandably, the conditional diffusion model also includes a local self-attention module; it performs feature extraction and fusion processing on multiple window features along the time axis, including: The local self-attention module extracts local features from each window feature to obtain local action features that correspond one-to-one with the sliding window. The query parameters used by the local self-attention module for each window feature are the same. The overlapping regions between two adjacent sliding windows are fused together according to the overlapping features in their respective local action features to obtain the fused features; The non-overlapping and overlapping features of the window features corresponding to each sliding window are combined according to the time axis to obtain continuous action features.

[0034] The division of window features can be implemented by a local self-attention module, or by adding a network layer to the local self-attention module and the conditional injection module. This application will not elaborate on these points in its embodiments.

[0035] This application does not limit how the fusion features are fused. In some embodiments, at least one of fixed weight averaging, distance-based weighted averaging, gating networks, confidence weights, or attention-based secondary fusion can be used. This application does not limit the specific methods used in this regard.

[0036] For example, the conditional diffusion model is used as the denoising network model below, referring to... Figure 2 The training process of the denoising network model in this application embodiment is described, wherein the arrows indicate the next step of the data flow. The specific steps are as follows: Step 1: Construct training sample skeletal action sequences, for example, such as... Figure 4 As shown, a clean 3D motion sequence with a preset number of frames is processed according to a preset noise step. By adding noise forward, a noisy 3D skeletal motion sequence is obtained.

[0037] Step 2: [The text appears to be incomplete and contains several typos. A more accurate translation would require the full context.] Noisy 3D skeletal motion sequences Noise step coding and target condition feature matrix data The input is fed into the conditional injection module, and the input embedding layer will... The motion vector of each frame is mapped as follows The 3D hidden vector is the dense feature vector of each frame within the time period corresponding to the 3D action input sequence, that is... The linear mapping layer performs a linear mapping on dense feature vectors to obtain linear mapping features. The temporal feature extraction layer extracts temporal features from the linear mapping features to obtain temporal position encoding features. In some embodiments, since the conditional diffusion model is a denoising network, feature extraction is also performed based on the noise step to obtain the noise step encoding. The temporal position encoding features, noise step, and linear mapping features can be fused into basic temporal features, such as... At this point, the basic time-series features are modulated based on the modulation parameters corresponding to all conditional features to obtain the action time-series feature sequence. It is noise step coding. It is a temporal position encoded feature (i.e., temporal position encoded feature), in which a linear mapping layer is used to project the skeletal motion representation of each frame onto a unified latent feature space. In some embodiments, the modulation parameters include at least one of scaling parameters and offset parameters; Step 3: Perform residual processing on the action timing features using the timing residual module.

[0038] Step 4: Perform feature extraction and fusion processing on the action temporal feature sequence output by the temporal residual module using the local self-attention module to obtain continuous action features; for example, refer to... Figure 3 As shown: Assume the action temporal feature sequence is a sequence of length T with a total of T frames. Assume that three window features are obtained based on the window length L and the sliding step size: window features corresponding to window 1, window features corresponding to window 2, and window features corresponding to window 3. Each window feature contains five frames of action temporal features. In this case, local self-attention is calculated for each window feature, and all three window features share the same set of shared query parameters. This yields local action features that correspond one-to-one with the window features. By fusing the overlapping regions of the local action features from two adjacent windows, the fused features for each frame can be obtained; for example, Figure 3 As shown, the fusion feature can be one of weighted fusion, gated fusion, and confidence fusion. In practical applications, the fusion strategy can be configured to be either weighted fusion, gated fusion, or confidence fusion according to the specific application scenario. This application embodiment will not elaborate on this point. For example, taking window 1 as an example, based on the fusion feature, the features of the 4th and 5th frames can be obtained. At this time, the frame features of each non-overlapping area in the local action features are combined with the fusion feature to obtain the T-frame action features, that is, continuous action features.

[0039] Here, we assume that the frame-level action features within a certain window are: Local self-attention is usually calculated as follows:

[0040] Wherein, assume the query parameter is , , Then different windows They all use the same Instead of using the first window The second window uses The third window uses If we further share the projection parameters for the Key and Value, we can also write it so that all windows share the same set. This enhances temporal consistency. For example, if a specific pose from a Dunhuang dance appears near frame 10 and frame 80, the model can recognize it using the same or similar rules, rather than interpreting it completely differently simply because it appears at different times. This allows for both local modeling for different window features and maintaining overall stylistic consistency. Compared to completely independent modeling for each window, which can lead to issues like a sharp wrist angle in one movement being smoothed out in another, or a large shoulder and hip twist in one window being weaker in another, the shared query parameters or shared query generation method in this application allow the model to reuse the same set of local movement recognition rules across all windows, thereby improving the consistency of similar local movements at different times.

[0041] Step 5: Process the continuous action features sequentially through the denoising backbone network and the action decoding layer to obtain the three-dimensional action prediction sequence; Step Six: Calculate the loss based on the 3D motion prediction sequence. The loss functions include a denoising loss function, a joint position loss function, a joint rotation loss function, a velocity smoothing loss function, and a foot slip constraint loss function. The denoising loss function determines the noise loss between the 3D motion prediction sequence and the desired clean motion sequence; the joint position loss function determines the joint position loss; the joint rotation loss function determines the rotation angle loss; the velocity smoothing loss function determines the velocity translation loss; and the foot slip constraint loss function determines the foot sliding position loss. At this point, based on these loss functions, predictions of noise, velocity, and the specific motion sequence can be achieved.

[0042] At this point, the total loss value can be obtained based on the loss values ​​calculated from each loss function, and then backpropagation can be performed based on the total loss value to update the model parameters of the conditional diffusion model.

[0043] Understandably, multimodal feature extraction from multimodal data yields initial conditional feature matrix data, including: Feature extraction is performed on the modal data of each modality in the multimodal data to obtain modal features that correspond one-to-one with the modality; The initial conditional feature matrix data is obtained by performing cross-modal semantic alignment on the features of each modality through a cross-modal semantic alignment network. The cross-modal semantic alignment network is configured with a text action loss function, an audio action synchronization loss function, and a visual skeleton similarity loss function.

[0044] During training, by configuring text action loss function, audio action synchronization loss function, and visual skeleton similarity loss function, the correspondence between text, audio, vision, and skeleton can be learned based on real dance samples. This allows for the learning of text semantics, music rhythm, visual style, and human kinematic constraints. Consequently, during cross-modal semantic alignment network inference, a 3D dance action sequence that simultaneously satisfies text semantics, music rhythm, visual style, and human kinematic constraints can be generated. Furthermore, feature balance can be achieved by aligning multiple modal features in a unified latent space under various modal input conditions.

[0045] In the embodiments of this application, the cross-modal semantic alignment network can be represented as an independently deployable model or as part of other models. In this regard, the embodiments of this application will not elaborate on each one.

[0046] For example, assuming the modalities of the multimodal data include text, audio (i.e., the aforementioned audio rhythm), images (pictures / videos), and skeletal prior data, then encoding can be performed separately by setting encoders that correspond one-to-one with each modality, such as... Figure 4 As shown, the text encoder extracts features from the text, the audio encoder extracts features from the audio, the visual encoder extracts features from the image / video, and the skeletal encoder extracts features from the skeletal prior data. At this point, it can be done as follows: Figure 4 As shown, modal features corresponding one-to-one with each mode can be obtained; For example, refer to Figure 4 As shown, the cross-modal semantic alignment network can achieve unified dimensional projection, shared latent space based on time axis alignment, and cross-modal alignment based on semantic similarity constraints. It also reconstructs the conditional feature matrix based on modal confidence / dynamic weights to obtain initial conditional feature matrix data. Unified dimensional projection transforms each modal feature into a modal feature of the same dimension through projection, allowing for comparison and fusion of these features within the same feature space. For example, a text encoder outputs action description semantic features, an audio encoder outputs beat, intensity, and musical phrase features, a visual encoder outputs pose contour and historical styling features, and a skeletal encoder outputs joint angles, velocity, contact state, and kinematic constraint features. These original features have different dimensions and meanings and cannot be directly added or input into a conditional diffusion model. They can be mapped to a unified dimension through linear layers, multilayer perceptrons, or Transformer projection layers. The latent feature vectors can be unified into 256-dimensional, 512-dimensional, or 768-dimensional features. After this processing, the "strong beat" in text, the strong beat in audio, the side-twisting contour in vision, and the shoulder reverse rotation relationship in skeleton can all be represented as conditional features in the same latent space, which can be uniformly called by subsequent models. Among them, the audio encoder can use audio convolutional networks, audio Transformers, or beat feature extraction networks to encode beat points, beat intensity, velocity curves, energy envelopes, and musical phrase structures into audio features. The visual encoder can use convolutional neural networks or visual Transformers to encode human contours, posture curves, arm overlaps, clothing occlusion, and historical styling features in images / videos into visual features; the skeletal encoder can use temporal convolutional networks, Transformers, or graph convolutional networks to encode three-dimensional joint coordinates, joint rotation, joint velocity, root node displacement, foot contact state, etc., into skeletal prior data.

[0047] For example, refer to Figure 4 As shown, it is assumed that text features are obtained through a text encoder during the training phase. Audio features are obtained through an audio encoder. Visual features are obtained through a visual encoder. Prior skeletal features are obtained via a skeletal encoder. . , , , Through trainable projection layers , , , Transformed into features of a unified dimension: , , , Among them, each They all reside in the same latent space and have the same feature dimensions. .

[0048] For example, refer to Figure 4As shown, during the training phase, text-action alignment loss (obtainable through the text-action loss function described above), audio-action synchronization loss (obtainable through the audio-action synchronization loss function), and visual-skeleton similarity loss (obtainable through the visual-skeleton similarity loss function) can be set to enable the cross-modal semantic alignment network to gradually learn the correspondences between different modalities. Specifically, the cross-modal semantic alignment network can learn the correspondences between different modalities through cross-attention, contrastive learning loss, similarity constraints, or gating fusion mechanisms. For example, for the same training sample, if the text description "flying theme, arms outstretched, turning and looking back," the corresponding audio rhythm, video frame, and 3D skeletal action should be close to each other in the latent space; while text, audio, or skeletal fragments unrelated to the sample should be far apart. During the training phase, one or more modalities can be used as input, and the remaining modalities as output. For example, when the model "sees the text" and "reverses the pipa," it not only obtains a language vector but can also align the language vector to the vicinity of the corresponding 3D action structure, arm posture, and body movement features to obtain the desired 3D action, visual features, or audio features. Similarly, when the audio beat is strong, it can also be aligned to the vicinity of temporal features such as action speed, center of gravity change, or limb extension. In some embodiments, a dynamic weighting mechanism can be introduced into the cross-modal semantic alignment network to address the problem of inconsistent quality between different modalities. For example, when the text description is more complete, the text modality has a higher weight; when the audio beat is clear, the audio modality has a higher weight; and when the input historical video quality is good and the confidence of key positions is high, the skeletal prior and visual modal weights are higher. For example, in the application phase of cross-modal semantic alignment networks, a weight prediction network outputs weights based on the confidence scores of each modality in the cross-modal semantic alignment network. , , , The initial condition matrix data is then normalized. It is assumed that the obtained condition feature matrix data is concatenated along the token dimension to form... Then based on weight , , , This allows us to obtain the fused conditional feature matrix data. .

[0049] This application does not restrict the form in which the target condition matrix data is input into the conditional diffusion model. It can be formed by sampling token concatenation or by time alignment, and the formation length is related to the length of the action to be generated. The corresponding conditional sequence, at this time It can be represented as or Each time point has corresponding rhythmic, posture, and skeletal conditions. Furthermore, non-frame-by-frame conditions such as text can be mapped to different time periods through copying, attention allocation, or switching based on action segments, thereby participating in the generation control of each frame or each local window.

[0050] The target conditional feature matrix data includes N modalities, and the initial conditional feature matrix data includes M modalities, where M is less than or equal to N. When the initial conditional feature data includes fewer than N modalities, the missing modal features are filled in using a pre-defined knowledge base. When a dynamic weighting mechanism is set, multimodal data can be input into the corresponding weight prediction network to obtain the weight of each of the N modalities. This allows the system to determine which condition to trust more based on the input quality and task requirements. For example, when the text explicitly specifies "turning around and looking back," the spatial pose constraint is stronger; when there is a downbeat in the music, the rhythm and action peak synchronization constraint is stronger; when the confidence of hand keypoints in historical vision is low, the weight of that local visual or skeletal observation can be reduced, relying more on the temporal consistency between consecutive frames and the prior knowledge of action primitives.

[0051] It is understood that modal features include at least one of text features, audio features, visual features, and skeletal prior features.

[0052] In practical applications, multimodal data does not necessarily have to simultaneously include text, audio, image / video, and skeletal data. For example, in some embodiments, the user may only input text, or text plus audio; in others, they may input historical video; and in still others, the user may only specify control information such as character, props, and style. In these cases, the system may first determine which modalities the user actually provided, then encode the existing modalities, and the resulting modal features are the features corresponding to the input modalities, thus obtaining the initial conditional matrix feature data. For information not provided by the user but still needed for action generation, it can be retrieved from pre-configured knowledge bases such as action primitives, style knowledge bases, or existing skeletal action databases, or a default prior or empty modal label can be used to obtain the target conditional matrix feature data C, which is then input into the diffusion generation model.

[0053] For example, taking a user input image or a Dunhuang mural design as an example, features such as human body contours, body direction, gestures, clothing styles, and stylistic features are extracted from the image as visual features. Then, similar action primitives, style tags, and skeletal templates are retrieved from a knowledge base (which can be a unified knowledge base, a knowledge base corresponding to each modality, or a knowledge base shared by multiple modalities). In this case, the condition matrix C mainly consists of visual stylistic features (i.e., the aforementioned visual features), action primitive features (i.e., the aforementioned text features), style tag features (i.e., the aforementioned audio features), and skeletal prior features. Since a single image does not have a complete time series, the system needs to supplement the continuous process of actions using the knowledge base and action priors.

[0054] For example, if the user specifies only a character or prop, such as multimodal data consisting of text information like "flying character," "holding a pipa," and "long sleeves," this information is first converted into text features such as character features, prop features, and motion constraint features. For instance, "holding a pipa" restricts the hand's position and the range of arm postures; "long sleeves" affects the range of motion, occlusion, and arm curves; and "flying character" corresponds to style tags like light, graceful, and flowing. In this case, the condition matrix C mainly consists of character features, prop features, style constraint features, and skeletal motion rules.

[0055] Understandably, text features are used to indicate at least one of the following motion attributes: action category, motion geometry, speed, and style; audio features include at least one of the following: rhythm features, audio velocity curve features, energy envelope features, and musical phrase embedding features; visual features are used to indicate at least one of the following: morphological contour features, posture curve features, and motion segment features; and skeletal prior features include at least one of the following: joint angle features, joint velocity features, contact state features, and motion feasibility features.

[0056] In some embodiments, for the skeletal prior features corresponding to the video, the body region of the action object (such as a dancer) in the video area can be determined first through a human detection / segmentation model, and then two-dimensional keypoints of each frame can be extracted through a two-dimensional pose estimation model. Then, the two-dimensional keypoints, body region features, and confidence scores of consecutive frames are input into a temporal three-dimensional pose estimation network to obtain the three-dimensional object pose. Finally, according to the application requirements, the three-dimensional object pose is represented as skeletal prior features such as parameterized human model parameters, three-dimensional joint coordinates under the standard skeletal hierarchy, joint rotation sequence, root node displacement, root node orientation, hand keypoints, and foot contact state. This makes it easy to stably detect two-dimensional keypoints from ordinary videos, while body region features can supplement occlusion and contour information. The three-dimensional object pose is used to recover realistic spatial movements, and the parameterized human model or standard skeletal hierarchy is used to ensure the rationality of human structure and provide a unified input for the subsequent conditional diffusion model to generate retargetable and editable three-dimensional skeletal motion sequences. For human detection / segmentation, object detection or instance segmentation models such as YOLO, Faster R-CNN, and Mask R-CNN can be used, or an integrated pose estimation model combining detection and pose estimation can be employed. Two-dimensional keypoints can be extracted using OpenPose, HRNet, ViTPose, MediaPipe Pose, or similar two-dimensional pose estimation networks. Two-dimensional keypoints refer to the two-dimensional coordinates and confidence scores of the main joints of the subject's body on the video image plane, such as head, neck, shoulder, elbow, wrist, hip, knee, ankle, toe, and heel keypoints. Further fine-grained keypoints such as finger joints and fingertips can also be included. The data format is typically the two-dimensional coordinates (ui,vi) of each joint in each frame, along with the corresponding confidence score pip_ipi, describing the projected position of the subject's pose on the image plane in that frame. Body region features are not simply coordinates of individual keypoints, but rather overall or local visual features extracted from the image region where the object is located. These include body bounding boxes, body contours, foreground masks, limb regions, clothing-occluded areas, sleeve areas, body orientation, torso contours, and overlapping arm regions. They can also be high-dimensional feature vectors extracted from cropped images by convolutional neural networks or visual Transformers. While two-dimensional keypoints primarily provide clues to joint positions, body region features supplement information that keypoints cannot fully express, such as clothing occlusion, limb overlap, body orientation, local contours, and the unique curved shapes characteristic of Dunhuang dance.

[0057] 3D object pose estimation is a spatial pose result further estimated based on 2D keypoints and body region features. 2D keypoints only represent the joint positions on the image plane and do not directly provide depth information, therefore they cannot fully represent 3D movements such as arm forward / backward relationships, torso twisting, and relative shoulder / hip rotation. The role of 3D object pose estimation is to combine temporal information from consecutive frames, 2D keypoint confidence, visual features of the body region, and kinematic constraints to elevate 2D joint positions into 3D joint coordinates or 3D joint rotations. Its output can be the coordinates (xi,yi,zi) of each joint in 3D space for each frame, or it can be the root node displacement, root node orientation, and local rotation angles of each joint. The temporal coding network mentioned here refers to a network that uses information from multiple consecutive frames for 3D pose estimation, such as temporal convolutional networks, Transformer temporal networks, recurrent neural networks, or state-space models. Its function is to use the continuity of actions between consecutive frames to correct occlusion and jitter errors in single-frame 2D detection.

[0058] The 3D pose can be fitted or converted into SMPL / SMPL-X parameters, or the 3D joint coordinates and joint rotations of the standard animation skeleton can be directly output. The common function of both is to form the "skeleton prior" that can be used by the subsequent diffusion model, that is, to provide kinematic information such as human structure, bone length, joint angle range, root node motion, hand posture and foot contact state.

[0059] Understandably, the method also includes: Obtain virtual character data; When virtual character data and target 3D motion sequence have the same skeletal topology, the virtual character data is isomorphically redirected according to the target 3D motion sequence to obtain 3D interactive data. When virtual character data and target 3D motion sequences have different skeletal topologies, heterogeneous redirection of virtual character data is performed to obtain 3D interactive data. Skeletal topology includes at least one of body proportions and the number of joints.

[0060] For example, refer to Figure 5 As shown, the application process of this application embodiment is as follows: Step 1: Generate a three-dimensional skeletal motion sequence with a time length of T (i.e., the target three-dimensional motion sequence in step S200). Step 2: Perform motion post-processing on the 3D skeletal motion sequence, which includes smoothing filtering, deduplication, and kinematic correction. Step 3: Output the 3D skeleton motion sequence after motion post-processing according to the preset output representation / format, such as BVH format, FBX format, or SMPL parameters; Step 4: Role Redirection like Figure 5 As shown, when the skeleton topology is consistent, isomorphic redirection is performed to directly map the source action and the target skeleton. like Figure 5 As shown, when the bone topology is different, heterogeneous redirection is performed, and the meta-action and target bone are heterogeneously mapped. The heterogeneous mapping includes scale mapping, joint constraints, topology video, and skinning weight adjustment. Step 5: Transfer the redirected skeletal motion of the character to the preset target character. The target character can then... Figure 5 The figures depicted include flying apsaras, bodhisattvas, powerful warriors, and virtual digital humans. Step 6: Output the target character to a preset application, such as 3D animation, virtual teaching, AR / VR display, and digital human-driven applications.

[0061] In some embodiments, this application also provides a terminal device, which includes a data acquisition module, a preprocessing module, a knowledge base management module, a pose estimation module, a cross-modal alignment module, a diffusion generation module, an overlapping window attention module, a temporal stabilization module, a redirection and rendering module, and a feedback update module.

[0062] The functions of each module are as follows: The data acquisition module is used to collect or receive Dunhuang dance text, audio, image / video and skeletal motion data.

[0063] The preprocessing module is used to perform data cleaning, frame rate unification, key frame extraction, beat detection, action primitive extraction, and coordinate normalization on the data acquired by the data acquisition module.

[0064] The knowledge base management module is used to maintain action primitives, style tags, prop status, character themes, theatrical events, and skeletal templates.

[0065] The pose estimation module is used to reconstruct 3D pose from images or videos and generate skeletal prior data that can be used for training or conditional guidance.

[0066] The cross-modal alignment module is used to map the multi-source features obtained from the preprocessing module to a shared latent space, and output a multi-condition feature matrix.

[0067] The diffusion generation module is used to perform forward noise modeling, backward denoising generation, and conditional action sampling.

[0068] The timing stabilization module is used to suppress jitter and error accumulation through residual connections, velocity / acceleration smoothing, and kinematic constraints.

[0069] The Overlapping Window Attention module is used to capture micro-action details within a local temporal window and to fuse overlapping window regions.

[0070] The Redirection and Rendering module is used to transfer the generated target 3D motion sequence to different virtual characters and output the visualization results.

[0071] The feedback update module is used to receive manual review and user feedback, and to update the dataset, rule base, and model parameters.

[0072] The overlapping window attention module represents the network architecture of the local self-attention module and is used to perform related operations of the local self-attention module. Unlike the global attention mechanism in existing technologies, the overlapping window attention module in this embodiment uses sliding windows that slide along the time axis with overlapping frames between adjacent sliding windows. Each sliding window employs self-attention computation to capture local action changes. Different sliding windows share query parameters or query generation methods to generate local action features, thereby improving the consistency of responses to similar local actions at different time points. Furthermore, the overlapping window attention module in this embodiment can fuse overlapping regions to ensure continuity in attitude, velocity, and acceleration in the outputs of adjacent sliding windows. Therefore, the model architecture of the overlapping window attention module in this embodiment can maintain sensitivity to local details while reducing the risk of completely severing information connections between windows. Compared to global attention, this structure reduces the computational overhead of long sequences, mitigates the risk of local action features being globally averaged, and reduces boundary mutations, thereby improving the continuity of actions.

[0073] The cross-modal alignment module characterizes the network architecture of the cross-modal semantic alignment network and is used to perform the steps related to the cross-modal semantic alignment network. In some embodiments, the cross-modal alignment module characterizes the mapping of text, audio, visual, and skeletal priors to a unified latent space. Specifically, the text side mainly describes action content and spatial pose, the audio side mainly describes rhythm and intensity variations, the visual side mainly provides historical form and artistic style priors, and the skeletal side mainly ensures motion feasibility. In this case, through cross-attention or gating fusion mechanisms and dynamic weights assigned to features of different modalities, a unified initial conditional feature matrix can be generated in the cross-modal alignment module.

[0074] In some embodiments, the terminal device includes a noise step embedding module, a condition injection module, and an action decoding module. The noise step embedding module is used to determine the noise step of the three-dimensional action input sequence with added noise. The condition injection module is used to conditionally modulate the multi-condition feature matrix (i.e., the target condition feature matrix data mentioned above) and the three-dimensional action input sequence so that the timing stabilization module can perform feature extraction and residual calculation based on the result of condition modulation.

[0075] The temporal stabilization module can preserve low-frequency continuous information in the input features, allowing the overlapping window attention module to focus on learning high-frequency details and conditional differences, thereby reducing the cumulative error during long sequence generation. Simultaneously, kinematic constraints represented by the target conditional feature matrix data can limit the generated results to meet basic human structural laws, including bone length stability, joint rotation range, foot contact, center of gravity changes, velocity smoothing, and acceleration smoothing.

[0076] The following reference Figure 6 The training and application process of the various models involved in the three-dimensional action sequence generation method of this application are described below, with specific steps as follows: Step 1: During the training phase, acquire multimodal data, for example, such as... Figure 6 As shown, by taking the target 3D motion sequence as the motion sequence related to Dunhuang dance, multimodal data related to Dunhuang dance can be obtained, including at least one of the following: professional dancer videos, rehearsal or performance audio, Dunhuang murals and statue images, dance textbook texts, descriptions of motion terminology, and existing 3D skeletal motion data. Among these, video data can be high-frame-rate, multi-view, or single-view high-definition video; audio data can include beat, tempo, dynamic variations, and musical phrase structure; and text data can include motion names, body descriptions, spatial orientation, prop information, and style categories.

[0077] Step 2: During the training phase, preprocessing is performed on the multimodal data. This preprocessing includes data cleaning and standardization. For example, in some embodiments, text may be sequentially segmented into words, entity recognition, and action primitive extraction, with normalized style labels set. Audio may be sequentially denoised, beat detection, segmentation, tempo estimation, and beat strength or energy strength analysis to obtain basic rhythmic information that can be subsequently encoded. This allows for the extraction of audio conditional features of a unified dimension, such as beat points, beat intensity, tempo curves, energy envelopes, and segment embeddings, based on the basic rhythmic information. In some embodiments, the basic rhythmic information may include audio tempo and... At least one of the speed curves; performing image / video denoising, frame rate unification, shot segmentation, body region localization, and keyframe filtering; for example, in some embodiments, for image / video data as a modal data, image denoising, frame rate unification, shot segmentation, body region localization, and keyframe filtering can be performed sequentially, and two-dimensional key points of each frame can be extracted based on the filtered key frames; thereby, the pose of the three-dimensional object can be obtained based on the two-dimensional key points, body region features, and confidence level, and further skeletal prior features can be formed; performing normalization processing such as skeletal hierarchy unification, coordinate system unification, frame rate resampling, and missing frame interpolation on existing skeletal data to obtain a skeletal sequence in a unified format.

[0078] Step 3: In the training phase, construct the action primitives and knowledge base: Based on the preprocessed data, build a knowledge base for Dunhuang dance movements, including a primitive base and a style knowledge base. For example, assign a motion_id to thematic movements such as flying apsaras, bodhisattvas, pipa musicians, drum musicians, lotus children, and strongmen warriors; assign an event_id to repertoire, musical segments, or action fragments; and assign a style_id to costumes, props, gestures, steps, spatial orientation, and character style. This knowledge base can then provide retrieval constraints for text semantics and action priors during the application phase.

[0079] Step 4: During the training phase, multimodal feature encoding is performed: semantic encoding is applied to the text description to obtain action category, spatial pose, direction, velocity, and style constraint vectors; acoustic encoding is applied to the audio to obtain beat points, beat intensity, velocity curves, energy envelopes, and musical phrase embeddings; visual encoding is applied to the image / video to obtain morphological contours, pose curves, and action segment features; temporal encoding is applied to the skeletal prior data to obtain joint angles, joint velocities, contact states, and motion feasibility features. The skeletal prior data can be obtained based on the input video. For example, for the input video, two-dimensional keypoints and body region features are first extracted, and then the three-dimensional object pose is estimated through a temporal encoding network to obtain skeletal prior data based on the three-dimensional object pose. A parametric body model or a standard skeletal hierarchy can be used to represent the body pose, and each frame's action is described by joint angles, root node displacement, root node orientation, hand keypoints, and contact states. In some embodiments, to reduce the impact of occlusion, temporal consistency loss, motion discrimination loss, and bone length constraint loss can be introduced into the temporal encoding network to correct keypoint offsets caused by limb self-occlusion. The feature extraction of different modalities can employ various methods... Figure 4 The encoder shown is used for encoding. In practical applications, the encoder can be an existing encoder that does not require training. In other embodiments, it can also be trained based on an existing encoder and the data preprocessed in step two. This application will not elaborate on these aspects.

[0080] Step 5: During the training phase, perform cross-modal semantic alignment: Map text, audio, visual, and skeletal prior features to the same latent space. Calculate the correspondence between different modalities using any one of cross-attention, gated fusion, or similarity constraints to obtain the aligned conditional features for each modality. Combine these conditional features to obtain the target conditional feature data C. In some embodiments, such as... Figure 6 As shown, modal confidence weights can be set, such as increasing the rhythm constraint weight when the input audio beat is clear, increasing the spatial pose constraint weight when the text action description is clear, and increasing the kinematic feasibility constraint weight when the skeletal prior quality is high. Based on the corrected weights, each conditional feature in the target conditional feature data is updated.

[0081] Step Six: Conditional Diffusion Model Training to Generate Conditional Diffusion Actions (i.e., Target 3D Action Sequences): For example, using a real or reconstructed 3D skeletal action sequence x0 as the training sample, and a denoising network model as the conditional diffusion model, forward noise is added according to a preset noise schedule to obtain xt at different noise steps. xt, noise step t, and target conditional feature data C are input into the denoising network model, and the noise, velocity, or clean action sequence is predicted for each output of the denoising network model. The model parameters of the denoising network model are adjusted based on a preset loss function, which may include a denoising loss function, joint position loss function, joint rotation loss function, velocity smoothing loss function, foot slip constraint loss function, text action alignment loss function, and music beat synchronization loss function. The real or reconstructed 3D skeletal action sequence x0 can represent the skeletal prior feature sequence obtained in Step Five; in other embodiments, it can also be a skeletal prior feature sequence directly collected from existing systems.

[0082] The conditional diffusion model includes overlapping window local self-attention calculation: an overlapping window local self-attention module is set in at least one layer of the denoising network. Temporal features of length T are divided into multiple local windows according to window length L and stride s, with overlapping frames between adjacent windows. Local self-attention is calculated within each window, and all windows share query generation parameters or query vectors to maintain consistent responses for the same local action patterns at different time points. Weighted averaging, gated fusion, or confidence-based fusion are used to output the overlapping regions, ensuring continuity of pose, velocity, and acceleration at the window boundaries.

[0083] The conditional diffusion model includes temporal residual processing to ensure temporal residual stability and kinematic constraints: temporal residual blocks are set in the denoising network, allowing input features to be fused with the denoised output through skip connections, reducing gradient decay and error accumulation during long sequence propagation. Furthermore, constant bone length constraints, joint angle range constraints, center of gravity balance constraints, ground contact constraints, and foot slip suppression constraints are applied to the generated sequence to ensure that the three-dimensional motion conforms to basic human kinematic laws.

[0084] At this point, based on the above steps, a trained conditional diffusion model can be obtained. In practical applications, steps one, two, four, and five can be referred to obtain initial conditional feature data that corresponds one-to-one with the input multimodal data. Then, based on the knowledge base constructed in step three, modal constraints are filled in on the initial conditional feature data to obtain target conditional feature data. The target conditional feature data and the preset three-dimensional action input sequence are input into the conditional diffusion model to obtain the target three-dimensional action sequence. At this point, steps seven and eleven can be executed for the output target three-dimensional action sequence.

[0085] Step 7: Motion Post-processing, Redirection, and Output: For example, the generated target 3D motion sequence is converted to BVH, FBX, SMPL parameters, or other 3D animation formats. In some embodiments, isomorphic redirection is performed for virtual characters with consistent skeletal levels; in other embodiments, heteromorphic redirection is performed for characters with different body proportions, skeletal topologies, or numbers of joints, and clipping and limb deformation are reduced through proportional mapping, joint constraints, and skinning weight fine-tuning. The final output is motion assets that can be used for 3D animation, virtual digital humans, AR / VR demonstrations, or interactive teaching.

[0086] Step 8: Manual Review and Incremental Feedback: Submit the generated results to professional dance teachers or reviewers for review, recording deviations in movement style, rhythm, posture errors, and visualization issues. Write the review results into a feedback database, incrementally updating the text vocabulary, movement primitives, style rules, and model parameters to improve the quality of subsequent generation. At this point, manual review and incremental feedback enable updates to the knowledge base and continuous optimization of the conditional diffusion model.

[0087] Therefore, the above embodiments of this application have at least the following effects: 1. When the method of this application embodiment is applied to the generation of three-dimensional motion sequences of Dunhuang dance, the local motion features of Dunhuang dance can be preserved more stably. Moreover, through the overlapping window local self-attention mechanism, the conditional diffusion model can focus on the microscopic changes of the wrist, fingertips, shoulders, hips, and torso within a narrow time range, reducing the weakening of cultural symbolic movements by global smoothing.

[0088] 2. The method in this application embodiment reduces the risk of action mutation at the window boundary by fusing the overlapping areas of adjacent windows and using a shared query mechanism. Therefore, combined with residual processing, it can improve the continuity of long sequence actions and reduce the accumulation of long sequence errors.

[0089] 3. The embodiments of this application can improve the stability of multimodal conditional control through cross-modal semantic alignment, and can also dynamically adjust the weight of each modality according to the input confidence, reducing the probability of semantic misalignment or rhythm misalignment caused by excessive dominance of a single modality.

[0090] 4. The embodiments of this application can construct bone length constraints, joint range constraints, foot contact constraints, and retargeting post-processing, making the output results more suitable for animation production, virtual teaching, AR / VR display, and digital human driving, thereby improving the engineering usability of motion assets.

[0091] 5. The embodiments of this application can also form a continuously optimized data closed loop, such as writing back the knowledge base and training set through manual review and user feedback, so as to gradually improve the generation quality in specific dance styles, specific roles and specific styles.

[0092] In summary, this embodiment of the application incorporates a local self-attention module with shared query parameters and an overlapping boundary fusion mechanism within the conditional diffusion model. This allows adjacent windows to have overlapping regions of Ls frames and maintains a consistent local feature extraction method during temporal shifts, reducing the risk of hard segmentation boundaries and minimizing computational overhead. Furthermore, this embodiment can fuse features from multiple modalities (such as text encoding vectors, audio rhythm vectors, style tag vectors, and skeletal prior vectors) into a conditional feature matrix data, which is then injected into each or a specified layer of the conditional diffusion model. This enables the conditional diffusion model to learn to reconstruct real action sequences from noisy action sequences during the training phase and to infer and generate action sequences that satisfy the conditional feature matrix data during the inference phase. Therefore, the 3D action sequences generated by this embodiment can balance sample diversity, action coherence, and style controllability.

[0093] Understandably, an electronic device provided according to embodiments of this application includes: At least one processor; At least one memory for storing at least one program; The above-described method for generating three-dimensional motion sequences is implemented when at least one program is executed by at least one processor.

[0094] For example, please refer to Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be a NAND flash, and the relevant program code is stored in the memory 402 and called by the processor 401 to execute the three-dimensional action sequence generation method of the present application embodiment; Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.

[0095] This application also provides a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor, it implements the control method of the above-described three-dimensional action sequence generation method.

[0096] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0097] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0098] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0101] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0102] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0103] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0104] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for generating a three-dimensional action sequence, characterized in that, The method includes: Multimodal feature extraction is performed on the multimodal data to obtain initial conditional feature matrix data, and target conditional feature matrix data is determined based on the initial conditional feature matrix data; the target conditional feature matrix data includes skeletal prior features and multimedia conditional constraint features; Based on the target condition feature matrix data, the target three-dimensional action sequence is obtained by performing three-dimensional action sequence reasoning on the three-dimensional action input sequence through the condition diffusion model. The three-dimensional action sequence reasoning is used to perform at least one round of action reasoning, and in each round of action reasoning, the conditional diffusion model performs the following steps: Temporal feature encoding is performed on the three-dimensional action input sequence and the target condition feature matrix data to obtain the action temporal feature sequence; According to a preset sliding step size, a sliding window of fixed length is used to sample the action timing feature sequence to obtain window features that correspond one-to-one with the sliding window; the length of the sliding step size is less than the length of the window; each sliding window is used to sample a single action unit; The multiple window features are extracted and fused according to the time axis, and the continuous action features of the fused process are used to predict the action sequence to obtain the three-dimensional action sequence of the current round of reasoning.

2. The method for generating three-dimensional action sequences according to claim 1, characterized in that, The conditional diffusion model includes a conditional injection module, which includes an input embedding layer, a linear mapping layer, and a temporal feature extraction layer. The step of performing temporal feature encoding on the three-dimensional action input sequence and the target conditional feature matrix data to obtain an action temporal feature sequence includes: The three-dimensional action input sequence is discretely conditionally vectorized through the input embedding layer to obtain the dense feature vector of each frame within the time period corresponding to the three-dimensional action input sequence. The linear mapping layer performs linear mapping transformation on each of the dense feature vectors to obtain linear mapping features. The temporal feature extraction layer extracts the temporal location features of the linear mapping features to obtain the temporal location encoding features. Based on the time position encoding features and the linear mapping features, the basic time series features are obtained; The basic temporal features are modulated based on the conditional features in the target conditional feature matrix data to obtain the action temporal feature sequence.

3. The method for generating three-dimensional action sequences according to claim 2, characterized in that, The conditional diffusion model further includes a temporal residual module. Before obtaining the window features corresponding one-to-one with the sliding window, the method further includes: The action timing features are updated by performing residual processing on the action timing features through the timing residual module.

4. The method for generating three-dimensional action sequences according to any one of claims 1 to 3, characterized in that, The conditional diffusion model further includes a local self-attention module; the feature extraction and fusion process for multiple window features according to the time axis includes: The local self-attention module extracts local features from each window feature to obtain local action features corresponding to each sliding window. The query parameters used by the local self-attention module for each window feature are the same. The overlapping areas between two adjacent sliding windows are fused together according to the overlapping features in their respective local action features to obtain fused features; The non-overlapping features and overlapping features of the window features corresponding to each sliding window are combined according to the time axis to obtain continuous action features.

5. The method for generating three-dimensional action sequences according to claim 1, characterized in that, The step of extracting multimodal features from multimodal data to obtain initial conditional feature matrix data includes: Feature extraction is performed on the modal data of each modality in the multimodal data to obtain modal features that correspond one-to-one with the modality; The initial conditional feature matrix data is obtained by performing cross-modal semantic alignment on each modal feature through a cross-modal semantic alignment network. The cross-modal semantic alignment network is configured with a text action loss function, an audio action synchronization loss function, and a visual skeleton similarity loss function.

6. The method for generating three-dimensional action sequences according to claim 5, characterized in that, The modal features include at least one of text features, audio features, visual features, and skeletal prior features.

7. The method for generating a three-dimensional action sequence according to claim 6, characterized in that, The text features are used to indicate at least one of the following motion attributes: action category, motion geometry, speed, and style; the audio features include at least one of the following: rhythm features, audio velocity curve features, energy envelope features, and musical phrase embedding features; the visual features are used to indicate at least one of the following: morphological contour features, posture curve features, and motion segment features; and the skeletal prior features include at least one of the following: joint angle features, joint velocity features, contact state features, and motion feasibility features.

8. The method for generating three-dimensional action sequences according to claim 1, characterized in that, The method further includes: Obtain virtual character data; When the virtual character data and the target 3D motion sequence have the same skeletal topology, the virtual character data is isomorphically redirected according to the target 3D motion sequence to obtain 3D interactive data. When the virtual character data and the target 3D action sequence have different skeletal topologies, the virtual character data is heterogeneously redirected to obtain 3D interactive data; The skeletal topology includes at least one of body proportions and the number of joints.

9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; The three-dimensional motion sequence generation method as described in any one of claims 1 to 8 is implemented when at least one of the programs is executed by at least one of the processors.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used by a processor to execute the three-dimensional action sequence generation method as described in any one of claims 1 to 8.

Images