A semantic-driven martial arts action synthesis method

Abstract

The present invention discloses a semantically driven martial arts action synthesis method, comprising: obtaining martial arts prompt text; synthesizing martial arts action segments based on the martial arts prompt text using a variable-duration action synthesis diffusion model, including: using a text semantic preprocessing network to predict text conditional features and action duration based on the martial arts prompt text; and generating noise action segments corresponding to the final noise scale as synthesized martial arts action segments based on the text conditional features and action duration through reverse diffusion iteration. In each single-step reverse diffusion, the text conditional features, action duration, current noise scale, and noise action segments obtained from the previous noise scale are used as input data, and a single-step reverse diffusion network is used to generate a rough clean action segment based on the input data. Then, forward diffusion is performed based on the rough clean action segment to obtain a noise action segment corresponding to the current noise scale. This method, driven by semantics, can achieve action synthesis and choreography with complex content, variable duration, and complex skills.

Description

Technical Field

[0001] The present invention belongs to the technical field of combining natural language with visual generation, and in particular relates to a semantic-driven martial arts action synthesis method. Background Art

[0002] The task of 3D human motion synthesis aims to use computers to synthesize natural 3D human animations, typically used to drive virtual characters or robots. This task is fundamental in the field of computer animation, yet also extremely challenging. Semantic-driven 3D motion synthesis aims to automatically extract natural language semantic information from text prompts, guiding the 3D motion synthesis process and achieving cross-modal motion semantic matching and synthesis. This method does not require the extensive labor and time required by manual methods, nor the complex ordinary differential equations required by physical methods, nor the strong reliance on camera-captured image or video data required by visual methods. Instead, it simply requires free text input to conveniently synthesize diverse motion content. Therefore, it is not difficult to foresee that this technology will significantly lower the threshold for motion data production and improve productivity in related industries in the future. It has broad application prospects in virtual reality, film and television games, robotic motion planning, and other fields.

[0003] Although semantically driven action synthesis tasks have attracted increasing attention from researchers, existing research work often suffers from the following three limitations:

[0004] 1) Lack of complex action-martial arts prompt text datasets: Currently, there are only three action-martial arts prompt text datasets available in the field, and all of these datasets contain a large number of simple daily behavioral movements such as walking, waving, and jumping. Therefore, the model can basically only learn to synthesize similar simple movements, and the synthesis quality of complex and professional action samples is poor.

[0005] 2) The duration of the synthesized action is fixed: Many semantically driven action synthesis methods tend to only focus on the synthesis quality of spatial pose, while ignoring the semantically driven estimation of action duration. This means that the action duration usually needs to be manually fixed for reasoning, resulting in a lack of diversity in the temporal dimension of the action.

[0006] 3) The synthesized results have a single action skill: Existing methods are usually trained on sample sets that only contain a single action skill. Therefore, it is difficult to synthesize temporal action choreography sequences that contain complex action skills, and there is insufficient attention to this issue in the field.

[0007] Therefore, there is an urgent need to leverage the advantages of existing semantically driven action synthesis technology while addressing the limitations of these areas. Currently, semantically driven action synthesis technology is still in its infancy. Limited by the lack of complex action-specific martial arts prompt text datasets, the synthesis results of existing methods often suffer from simple action content, fixed action duration, and limited skill sets.

[0008] Martial arts movements are a collection of complex, specialized skills, and are widely used in fields such as film, television, animation, and video games. However, because modeling martial arts movements often requires additional expertise from animators or the use of motion capture by martial arts practitioners, acquiring high-quality martial arts movement data is significantly more expensive than for simple movements like walking, waving, and jumping. Furthermore, martial arts movements often lack fine-grained, long text describing a specific set of moves, which poses a challenge in constructing matching text annotation datasets. Consequently, there are currently no action-based martial arts prompt text datasets specifically designed for martial arts movements. Summary of the Invention

[0009] In view of the above, the purpose of the present invention is to provide a semantic-driven martial arts action synthesis method, which can realize the synthesis and arrangement of actions with complex content, variable duration, and compound skills through semantic driving.

[0010] To achieve the above-mentioned object of the invention, an embodiment provides a semantic-driven martial arts action synthesis method, comprising the following steps:

[0011] Obtain the martial arts prompt text corresponding to the martial arts action clip to be synthesized;

[0012] Martial arts action segments are synthesized based on martial arts prompt text using a variable duration action synthesis diffusion model, including: using a text semantic preprocessing network to predict text conditional features and action duration based on the martial arts prompt text, and generating noise action segments corresponding to the final noise scale as synthesized martial arts action segments based on the text conditional features and action duration through inverse diffusion iteration;

[0013] Among them, in each single-step reverse diffusion, the text conditional features, action duration, current noise scale and the noise action fragments obtained at the current noise scale are used as input data. The single-step reverse diffusion network is used to generate rough and clean action fragments based on the input data, and forward diffusion is performed based on the rough and clean action fragments to obtain the noise action fragments corresponding to the next noise scale.

[0014] Preferably, the text semantic preprocessing network includes a text semantic conditional encoder and an action duration prediction network, wherein the text semantic conditional encoder is used to encode the martial arts prompt text to obtain text conditional features, and the action duration prediction network is used to predict the action duration based on the martial arts prompt text, and the action duration is used to constrain the number of frames of the martial arts action.

[0015] Preferably, the text semantic conditional encoder includes a CLIP pre-training model, and the martial arts prompt text is encoded using the CLIP pre-training model to obtain text conditional features;

[0016] The action duration prediction network includes a first linear layer, a BiGRU, a first MLP and a Softmax activation layer connected in sequence. The input martial arts prompt text passes through the first linear layer, BiGRU, the first MLP and the Softmax activation layer in sequence to obtain a discrete duration probability distribution, and the action duration is sampled from the discrete duration probability density.

[0017] Preferably, the single-step inverse diffusion network includes a second linear layer, a second MLP, a third linear layer, a Transformer encoder, and a fourth linear layer. After the second linear layer, the second MLP, and the third linear layer respectively map the noisy action fragment obtained by the current noise scale, the current noise scale, and the text conditional features, the three mapping results are spliced ​​with the result of position encoding of the action duration to obtain a splicing vector. The splicing vector is sequentially encoded by the Transformer encoder and linearly mapped by the fourth linear layer to obtain a rough and clean action fragment.

[0018] Preferably, the method further comprises: constructing an autoregressive diffusion model based on the variable-duration action synthesis diffusion model, wherein the autoregressive diffusion model comprises a priori condition encoder and the variable-duration action synthesis diffusion model;

[0019] Based on the regression diffusion model, multiple martial arts action clips are generated for multiple martial arts prompt texts through autoregressive iteration, and the multiple martial arts action clips are synthesized into a coherent action clip, specifically including:

[0020] In each autoregressive iteration step, the historical martial arts prompt text corresponding to the previous autoregressive iteration step, the historical martial arts action fragment corresponding to the historical martial arts prompt text, and the current martial arts prompt text corresponding to the current autoregressive iteration step are used as inputs of the regression diffusion model. The prior condition encoder is used to encode the historical martial arts prompt text and the corresponding historical martial arts action fragment to generate a historical prior condition vector. The text semantic preprocessing network is used to generate text condition features and action duration based on the current martial arts prompt text. Based on the text condition features, action duration and the historical prior condition vector, a noise action fragment corresponding to the final noise scale is generated through inverse diffusion iteration as the martial arts action fragment corresponding to the current autoregressive iteration step.

[0021] Preferably, the prior condition encoder includes a prior text encoder and a prior action encoder. The prior text encoder encodes the historical martial arts prompt text to obtain historical text conditional features, and the prior action encoder encodes the historical martial arts action fragments to obtain historical action features. The historical text conditional features and historical action features are spliced ​​to obtain a historical prior condition vector.

[0022] Preferably, the prior text encoder includes a CLIP pre-training model, and the CLIP pre-training model is used to encode the historical martial arts prompt text to obtain historical text conditional features;

[0023] The prior action encoder includes a fifth linear layer, and the fifth linear layer is used to encode historical martial arts action clips to obtain historical action features.

[0024] Preferably, the method of generating a noise action segment corresponding to the final noise scale as the martial arts action segment corresponding to the current autoregressive iteration step through reverse diffusion iteration based on the text conditional features, the action duration, and the historical prior condition vector comprises:

[0025] Taking text conditional features, action duration, historical prior condition vector, current noise scale, and noisy action fragments obtained at the current noise scale as input data, a single-step inverse diffusion network is used to generate rough clean action fragments based on the input data. Specifically, the second linear layer, the second MLP, and the third linear layer respectively map the noisy action fragments obtained at the current noise scale, the current noise scale, and the text conditional features. The three mapping results, the historical prior condition vector, and the result of position encoding of the action duration are then concatenated to obtain a concatenated vector. The concatenated vector is then encoded by the Transformer encoder and linearly mapped by the fourth linear layer to obtain a rough clean action fragment.

[0026] Based on the rough clean action fragment, forward diffusion is performed to obtain the noise action fragment corresponding to the next noise scale, and then the noise action fragment corresponding to the final noise scale is generated through iterative reverse diffusion as the martial arts action fragment corresponding to the current autoregressive iteration step.

[0027] Preferably, the method further comprises constructing an action-martial arts prompt text dataset, and specifying martial arts prompt texts of the martial arts action segments to be synthesized based on the martial arts action-martial arts prompt text dataset.

[0028] Preferably, the method also includes visual display of martial arts action clips, including temporal visualization and spatial visualization, wherein the temporal visualization provides a playback progress bar to perform temporal animation playback of the action sequence of the martial arts action clips and jump to specified time frames; the spatial visualization provides visualization of the three-dimensional posture of each frame in two modes: skeleton and grid.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] To address the problems of fixed action duration and single action skills in the synthesis results of existing methods, a martial arts action choreography method combining action duration prediction was designed and implemented. The action duration prediction network adaptively infers semantic action duration from martial arts prompt text input, and then determines the number of action frames required in the subsequent martial arts action synthesis process, improving the temporal dimension diversity of existing methods; the autoregressive encoding model can use the encoding of the previous synthesized action segment to guide the synthesis of the next action segment, thereby autoregressively iteratively synthesizing a coherent temporal action choreography sequence containing complex action skills. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 is a flow chart of a semantic-driven martial arts action synthesis method provided in an embodiment;

[0033] Figure 2 This is the structure and flow chart of the variable duration action synthesis diffusion model provided by the embodiment;

[0034] Figure 3 Schematic diagram of the structure of the action duration prediction network provided by the embodiment;

[0035] Figure 4 1 is a schematic structural diagram of a single-step reverse diffusion network provided in an embodiment;

[0036] Figure 5 This is a comparison diagram of the visualization results of synthesizing martial arts action clips using the variable duration improvement solution provided in the embodiment, including the annotated martial arts action clip durations;

[0037] Figure 6 It is the structure and flow chart of the autoregressive diffusion model provided by the embodiment;

[0038] Figure 7 is a schematic structural diagram of a priori condition encoder provided in an embodiment;

[0039] Figure 8 This is a flow chart of generating rough cleaning action segments by a single-step inverse diffusion network in an autoregressive diffusion model provided by an embodiment;

[0040] Figure 9: This is a comparison chart of the visualization results of the coherent action clips generated by the improved autoregressive solution provided in the embodiment, where the horizontal line represents the distorted part of the baseline solution;

[0041] Figure 10 A flowchart of constructing a martial arts action-semantic text dataset provided in the embodiment;

[0042] Figure 11 It is a structural diagram of the martial arts action synthesis prototype system provided in the embodiment;

[0043] Figure 12 This is a design diagram of the user interaction interface in the system provided by the embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0045] The embodiment provides a semantic-driven martial arts action synthesis method, wherein semantic-driven martial arts action synthesis, also known as text-driven action synthesis or text-described action generation, aims to input one or more paragraphs of free text, synthesize and output a three-dimensional human action sequence that conforms to the semantics of the text description.

[0046] like Figure 1 As shown, the semantic-driven martial arts action synthesis method provided in the embodiment includes the following steps:

[0047] Step 1: Obtain martial arts prompt text corresponding to the martial arts action clip to be synthesized.

[0048] In an embodiment, the martial arts prompt text is a text containing semantic information input by a user, and the martial arts prompt text serves as driving information to guide the synthesis of martial arts action segments. For example, the martial arts prompt text may be "a person is turning."

[0049] Step 2: Utilize the variable duration action synthesis diffusion model to synthesize martial arts action clips based on martial arts prompt text.

[0050] In the embodiment, the variable duration action synthesis diffusion model can generate martial arts action clips driven by martial arts prompt text based on the diffusion process. Figure 2As shown, the variable-duration action synthesis diffusion model includes a text semantic preprocessing network, a single-step inverse diffusion network, and an iterative inverse diffusion process. The text semantic preprocessing network includes a text semantic conditional encoder and an action duration prediction network. The text semantic conditional encoder is used to encode the martial arts prompt text to obtain text conditional features. The text semantic conditional encoder can use the CLIP pre-trained model. This CLIP pre-trained model has established a high-quality text-image semantically aligned latent feature space through contrastive learning using a large-scale text-image dataset. This latent feature space can be used to convert the martial arts prompt text into text conditional features.

[0051] The action duration prediction network uses probability density estimation to adaptively predict action duration based on the martial arts prompt text as a driver. This action duration is used to determine the number of frames required for the subsequent action synthesis process, that is, the number of frames constraining the martial arts action. The action duration prediction network can use a classifier network, and its discrete duration probability density estimate is:

[0052] p(L|S)={p(L min |S),p(L min +1|S),…,p(L max |S)}

[0053] Therefore, the loss function is designed as the cross entropy loss of the multi-classification task, and the formula is defined as:

[0054]

[0055] Among them, l i is the one-hot encoding of the true duration, if and only if the duration is i l i =1, otherwise l i =0, S represents the martial arts prompt text, L represents the duration, and p(i|S) represents the probability density of generating duration i under the drive of the martial arts prompt text S.

[0056] Due to the diversity of action clip durations, it is more reasonable to model the mapping of martial arts prompt text to action duration as a density estimation problem rather than directly regressing the specific numerical value of the action duration corresponding to the text. This approach not only achieves semantically driven adaptive estimation of action duration, but also ensures the diversity of action duration.

[0057] The embodiment preferably provides Figure 3 The action duration prediction network shown in the figure includes a first linear layer, a BiGRU, a first MLP, and a Softmax activation layer connected in sequence. The input martial arts prompt text passes through the first linear layer, the BiGRU, the first MLP, and the Softmax activation layer in sequence to obtain a discrete duration probability distribution, and the action duration is sampled from the discrete duration probability density.

[0058] The single-step inverse diffusion network provided in the embodiment is designed to perform a Markov random decision process. The algorithm randomly samples the noise action segment X with the maximum noise scale from the standard normal distribution N(0,I). t , perform a reverse diffusion process, specifically according to the text condition feature c, action duration F, current noise scale t and current noise scale X t The obtained noise action segment X t To generate rough cleaning action clips And forward diffusion is performed based on the rough cleaning action fragment to obtain the noise action fragment X corresponding to the next noise scale t-1 .

[0059] Single-step reverse diffusion process is X t to X t-1 The transfer process, its transfer strategy needs to use the network with parameter θ to learn the sampling distribution. In this embodiment, instead of using the “score prediction method”, we choose to directly predict the rough cleaning action segment samples. This method is also called "signal prediction method" and its specific formula is defined as (the symbol: = indicates the definition):

[0060]

[0061] in, represents a neural network with parameters θ, receiving a noisy action segment X t and the current noise scale t as input, predict based on The noise action segment X at time t-1 is obtained by forward diffusion t-1 , the above formula only involves the unconditionally guided single-step inverse diffusion process, while the conditionally guided single-step inverse diffusion process formula used is defined as:

[0062]

[0063] c represents the text conditional feature vector extracted in the previous step, and c and X t ,t,F are sent into the single-step reverse diffusion backbone network Performing rough cleaning action clips The prediction synthesis, and the reverse diffusion iterative process will repeatedly use the single-step reverse diffusion backbone network to iteratively denoise, thereby restoring the rough clean action fragments. According to the Markov random decision process, the algorithm will randomly sample the noise action fragment X with the maximum noise scale from the standard normal distribution N(0|I) T , iteratively perform a single-step inverse diffusion process, each time predicting a rough clean action segment Then forward diffusion is performed to obtain the noise action segment X of the next noise scalet-1 , until the noise scale is 0, the original cleaning action segment X0 is restored, and the calculation formula is further summarized as follows:

[0064]

[0065] Each step P in the formula θ (X t-1 |X t ) require independent sampling for iteration, and there is no closed-form one-step sampling method. The designed single-step inverse diffusion backbone network training loss function is defined as:

[0066]

[0067] The embodiment preferably provides Figure 4 The single-step inverse diffusion network shown in the figure includes a second linear layer, a second MLP, a third linear layer, a Transformer encoder, and a fourth linear layer. The second linear layer, the second MLP, and the third linear layer respectively generate the noise action fragment X obtained at the current noise scale. t After mapping the current noise scale t and the text conditional feature c, the three mapping results are spliced ​​with the result of position encoding (PE) of the action duration F to obtain a splicing vector, which is then encoded by the Transformer encoder and linearly mapped by the fourth linear layer to obtain a rough clean action segment.

[0068] Based on the above text semantic preprocessing network and single-step inverse diffusion network, as follows Figure 2 The reverse diffusion iterative process shown performs reverse diffusion to generate a noise action segment X0 corresponding to the final noise scale as a synthesized martial arts action segment.

[0069] In the embodiment, the martial arts action clips synthesized based on the variable duration action synthesis diffusion model are referred to as the variable duration improvement scheme. The final action synthesis visualization results of the variable duration improvement scheme and various benchmark schemes on the HumanMartialArts dataset are shown as follows: Figure 6 As shown in FIG, each action segment is sampled at intervals of 8 frames.

[0070] First, the synthesized motions of the MDM benchmark are essentially consistent with the ground-truth motions, making it the solution with the best synthesis quality in the visualization results. Second, the synthesized motions of the variable duration improvement solution are very similar to both the MDM baseline and the ground-truth, making it the solution with the second-best synthesis quality. Because the duration prediction network estimates a longer duration than the ground-truth duration, the overall motion speed of the variable duration improvement solution is slower than the MDM baseline. However, the synthesized additional motions remain realistic and natural, demonstrating temporal diversity. Furthermore, the synthesized motions of the T2M benchmark also synthesize longer motion segments due to the duration prediction network, but the final result is severely distorted. While the visualization results match the semantics of the outward swing leg motion, they are clearly substandard and stiff compared to the ground-truth motion. Finally, the MoitonDiffuse benchmark uses ground-truth duration for synthesis, resulting in a lack of temporal diversity. While the synthesis quality is better than the T2M baseline, and the semantics of the outward swing leg motion are largely consistent, there is a significant difference compared to the ground-truth motion. This may explain why the FID metric of the MoitonDiffuse benchmark is poor, while the Diversity and Multimodality diversity metrics are good.

[0071] In summary, the qualitative and quantitative results are basically consistent. The MDM benchmark is currently the optimal solution for semantically driven action synthesis in complex action domains. The variable duration improvement scheme can further improve the temporal diversity of synthesized actions while maintaining the performance of the MDM benchmark.

[0072] The variable duration improvement scheme provided by the above embodiment can realize the generation of a martial arts action segment based on a martial arts prompt text. The temporal action choreography task requires that the model can generate a series of action segments that conform to the corresponding semantics based on a series of text prompts describing different action skills, and the transition between the action segments is natural and smooth. Based on this, the embodiment constructs an autoregressive diffusion model based on the variable duration action synthesis diffusion model, and generates a series of martial arts prompt texts S based on the autoregressive diffusion model through autoregressive iteration. 1:N Generate multiple martial arts action clips that are semantically consistent and smoothly transitioned And combine multiple martial arts action clips into a coherent action clip.

[0073] like Figure 6 As shown, the autoregressive diffusion model includes a priori condition encoder and a variable duration action synthesis diffusion model, wherein the priori condition encoder is based on the historical martial arts prompt text S corresponding to the previous autoregressive iteration step i-1. i-1 Historical martial arts action clips corresponding to historical martial arts prompt texts Predict output history prior condition vector z past , the z pastUsed to guide the autoregressive iteration step i to synthesize the current action segment with smooth transition

[0074] like Figure 7 As shown, the embodiment preferably provides a priori condition encoder including a priori text encoder and a priori action encoder. The priori text encoder can select a CLIP pre-training model, and the CLIP pre-training model is used to train the historical martial arts prompt text S i-1 Encode to obtain historical text conditional feature c past , the prior action encoder can choose the fifth linear layer, and use the fifth linear layer to perform the historical martial arts action clips. Encoding historical action features Then, historical text conditional features and historical action features are concatenated to obtain the historical prior conditional vector z past .

[0075] The text semantic preprocessing network included in the variable duration action synthesis diffusion model also uses the text semantic conditional encoder and action duration prediction network in the autoregressive process according to the current martial arts prompt text S i Generate text conditional feature c i , action duration F i .

[0076] The autoregressive diffusion model is essentially a method for predicting martial arts movements. Combined with autoregressive logic, it can naturally complete the task of temporal action choreography. After being encoded by the prior condition encoder and predicted by the text semantic preprocessing network, the historical prior condition vector z past and text condition feature c i , action duration F i , the current noise scale t and the noise action fragment obtained by the current noise scale As input data, such as Figure 8 As shown, a single-step inverse diffusion network is used to generate rough cleaning action fragments based on the input data. Specifically, the second linear layer, the second MLP layer, and the third linear layer respectively obtain the noise action fragments of the current noise scale. Current noise scale t, text conditional feature c i After mapping, the three mapping results z m 、z t 、z c , z t 、z c and z past Concatenate to form the guidance field latent vector z tk , the z tk With z m And the action duration F iThe results of position encoding (PE) are concatenated to obtain a concatenated vector, which is then encoded by the Transformer encoder and linearly mapped by the fourth linear layer. Rough cleaning action clip. Then based on the rough cleaning action clip Perform forward diffusion to obtain the noise action fragment corresponding to the next noise scale Then, the final noise scale (t=0) action segment is generated through iterative inverse diffusion As the martial arts action clip of the current autoregressive iterative step

[0077] Based on the above autoregressive diffusion model, the autoregressive iterative process is as follows: Figure 7 The right side of the dotted line shows the first section of the martial arts prompt text S 1 At the beginning, the corresponding martial arts action clips were synthesized by continuously iteratively using the autoregressive diffusion model Finally, the temporal action choreography results of all action clips are generated, that is, the temporal action choreography sequence Therefore, the loss function of the single-segment variable-duration action synthesis diffusion model can be used to derive the mathematical representation of the overall loss function after the autoregressive diffusion model is updated:

[0078]

[0079] This formula is also based on the overall loss function representation of the data distribution. In actual practice, a single loss function backpropagation requires two forward processes. Finally, the training loss function used in the autoregressive diffusion model network in practice is defined as:

[0080]

[0081] The loss function L train Although there is no explicit definition of the coherence loss term for smooth transition, due to and The ground truth action clips themselves are smooth and coherent, so the synthetic action and As long as you get closer and This is sufficient to implicitly learn the coherence knowledge of the ground truth and synthesize smoothly transitioned temporal action choreography sequences.

[0082] In the embodiment, the martial arts action clip synthesis based on the autoregressive diffusion model is referred to as the autoregressive improved scheme. The action synthesis visualization results of the autoregressive improved scheme and various benchmark schemes on the HumanMartialArts dataset are as follows: Figure 9As shown in the figure. Due to the long duration of the two consecutive action segments, each of the following action sequences is sampled every 10 frames. In terms of synthesis quality, the autoregressive solution is closest to the ground truth visualization results; in terms of motion transition, the autoregressive solution is the most natural and smooth; and in terms of semantic matching, both the preceding and following actions of the autoregressive solution best match the corresponding textual prompt semantics. Therefore, the autoregressive solution is the best solution in the qualitative evaluation.

[0083] The semantically driven martial arts action method provided in the above embodiment can achieve semantically driven synthesis and sequential action choreography of martial arts action segments with complex content, variable duration, and complex skills. To this end, the embodiment also constructed a martial arts action-semantic text dataset (HumanMartialArts) containing complex professional skills to promote the autoregressive diffusion model to learn complex professional action knowledge. Based on the constructed HumanMartialArts dataset, martial arts prompt text related to martial arts actions is input, and the above method is used to infer and output three-dimensional martial arts action sequence segments.

[0084] The HumanMartialArts dataset not only expands the data volume of the original action sequence segmentation and refinement, but also annotates each refined martial arts action segment with a variety of text prompt annotations. The data format of HumanMartialArts will be processed and generated with reference to the format of the HumanML3D dataset, and is additionally equipped with coherence information of action sequences similar to that in the BABEL dataset. Therefore, the HumanMartialArts dataset can not only naturally adapt to the existing semantic-driven action synthesis model, but also support the research and training of sequential action choreography tasks. Figure 10 As shown in the figure, the construction process of the HumanMartialArts dataset is mainly divided into the following five steps:

[0085] The construction process is as follows Figure 10 As shown,

[0086] 1) Establishment of a core lexicon for martial arts movements: Based on the professional martial arts movements described in the book "Film and Television Martial Arts Movement Skills", we break down the movement skills, summarize all relevant martial arts movement skill terms, and establish a multi-granular core lexicon for martial arts movements.

[0087] 2) Hand shape category label annotation: Based on the statistically summarized core vocabulary of martial arts movements, each martial arts movement skill is annotated with a hand shape category label to distinguish the specific fist, palm and finger forms.

[0088] 3) Action data refinement: The original motion capture action sequences are manually divided and refined according to the corresponding martial arts action skills in the core vocabulary to obtain more refined martial arts action clips of appropriate duration, and finally the number of samples is further expanded through data enhancement.

[0089] 4) Action segment coherence annotation: Based on the segment division results of the action data, an index relationship dictionary between all coherent action segments is established.

[0090] 5) Text prompt description sentence generation: Finally, all refined martial arts action fragments and their corresponding martial arts action skill terms are used to automatically generate matching text prompt description sentences and corresponding part-of-speech tags.

[0091] In order to verify the practical feasibility of the method proposed in this invention, the technical content and front-end visualization interface of the method were integrated, and a semantic-driven martial arts action synthesis prototype system based on a diffusion model was designed and developed for martial arts action data. The system supports users to input martial arts prompt text or action labels, and then execute the above-mentioned martial arts action synthesis method. The final synthesis result will be represented by a skeleton model or a mesh model for spatiotemporal three-dimensional visualization for users to check or save. The overall architecture design of the semantic-driven martial arts action synthesis prototype system is as follows: Figure 11 As shown in the figure, the system architecture is mainly divided into three layers: data storage layer, business logic layer, and visual interaction layer.

[0092] The data storage layer is used to store 3D motion sequence data and provides communication support for data reading and writing with the business logic layer. This allows the system to store and read generated 3D motion sequences. This layer is designed to store data in local disk space, using the SMPL+H format and stored in the npy file format.

[0093] The business logic layer primarily focuses on the core martial arts action synthesis method and executes corresponding business logic based on user interactions in the visual interaction layer. The business logic layer communicates with the visual interaction layer via messages to obtain the input data and parameters required by the core method, executes the network inference process, and returns the inference results to the visual interaction layer for subsequent user operations.

[0094] The visual interaction layer is the level at which users conduct human-computer interaction, and will provide a convenient and effective visual interaction interface for users to complete all related operations. Figure 12 As shown, the key operations include: user input, access actions, action synthesis, and action visualization.

[0095] For user input functions, the interactive interface will provide a text input panel and a check list of martial arts action tags. Users can freely enter and edit text prompts or check action tags.

[0096] For the access action function, the interactive interface will provide a menu bar button for access actions, which is used to save and read action sequences. The read action sequences can be directly visually checked.

[0097] For the action synthesis function, the interactive interface will provide a synthesis button for three-dimensional action sequences. After the user enters the required data, clicking the synthesis button will cause the business logic layer to execute the core algorithm to realize the synthesis of martial arts action sequences.

[0098] Regarding action visualization, the interactive interface will provide spatiotemporal visualization of 3D action sequences, primarily divided into temporal and spatial visualization. Temporal visualization primarily provides a progress bar for playing back the action sequence in timed animation and jumping to specified time frames. Spatial visualization will provide both skeletal and mesh model visualizations of each frame's 3D pose, with mesh visualization using SMPL+H representation.

[0099] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A semantic-driven martial arts action synthesis method, characterized in that: The following steps are involved: Obtain the martial arts prompt text corresponding to the martial arts action clip to be synthesized; Martial arts action segments are synthesized based on martial arts prompt text using a variable duration action synthesis diffusion model, including: using a text semantic preprocessing network to predict text conditional features and action duration based on the martial arts prompt text, and generating noise action segments corresponding to the final noise scale as synthesized martial arts action segments based on the text conditional features and action duration through inverse diffusion iteration; Among them, in each single-step reverse diffusion, the text conditional features, action duration, current noise scale and the noise action fragments obtained at the current noise scale are used as input data. The single-step reverse diffusion network is used to generate rough and clean action fragments based on the input data, and forward diffusion is performed based on the rough and clean action fragments to obtain the noise action fragments corresponding to the next noise scale.

2. The semantic-driven martial arts action synthesis method according to claim 1, characterized in that: The text semantic preprocessing network includes a text semantic conditional encoder and an action duration prediction network, wherein the text semantic conditional encoder is used to encode the martial arts prompt text to obtain text conditional features, and the action duration prediction network is used to predict the action duration based on the martial arts prompt text, and the action duration is used to constrain the number of frames of the martial arts action.

3. The semantic-driven martial arts action synthesis method according to claim 2, characterized in that: The text semantic conditional encoder includes a CLIP pre-training model, and the martial arts prompt text is encoded using the CLIP pre-training model to obtain text conditional features; The action duration prediction network includes a first linear layer, a BiGRU, a first MLP and a Softmax activation layer connected in sequence. The input martial arts prompt text passes through the first linear layer, BiGRU, the first MLP and the Softmax activation layer in sequence to obtain a discrete duration probability distribution, and the action duration is sampled from the discrete duration probability density.

4. The semantic-driven martial arts action synthesis method according to claim 1, characterized in that: The single-step inverse diffusion network includes a second linear layer, a second MLP, a third linear layer, a Transformer encoder, and a fourth linear layer. The second linear layer, the second MLP, and the third linear layer respectively map the noisy action fragment obtained by the current noise scale, the current noise scale, and the text conditional features. The three mapping results are spliced ​​with the result of position encoding of the action duration to obtain a splicing vector. The splicing vector is sequentially encoded by the Transformer encoder and linearly mapped by the fourth linear layer to obtain a rough and clean action fragment.

5. The semantic-driven martial arts action synthesis method according to claim 4, characterized in that: Also includes: An autoregressive diffusion model is constructed based on the variable-duration action synthesis diffusion model. The regression diffusion model includes a priori condition encoder and a variable-duration action synthesis diffusion model. Based on the regression diffusion model, multiple martial arts action clips are generated for multiple martial arts prompt texts through autoregressive iteration, and the multiple martial arts action clips are synthesized into a coherent action clip, specifically including: In each autoregressive iteration step, the historical martial arts prompt text corresponding to the previous autoregressive iteration step, the historical martial arts action fragment corresponding to the historical martial arts prompt text, and the current martial arts prompt text corresponding to the current autoregressive iteration step are used as inputs of the regression diffusion model. The prior condition encoder is used to encode the historical martial arts prompt text and the corresponding historical martial arts action fragment to generate a historical prior condition vector. The text semantic preprocessing network is used to generate text condition features and action duration based on the current martial arts prompt text. Based on the text condition features, action duration and the historical prior condition vector, a noise action fragment corresponding to the final noise scale is generated through inverse diffusion iteration as the martial arts action fragment corresponding to the current autoregressive iteration step.

6. The semantic-driven martial arts action synthesis method according to claim 5, characterized in that: The prior condition encoder includes a prior text encoder and a prior action encoder. The prior text encoder encodes the historical martial arts prompt text to obtain historical text condition features. The prior action encoder encodes the historical martial arts action fragments to obtain historical action features. The historical text condition features and the historical action features are spliced ​​to obtain a historical prior condition vector.

7. The semantic-driven martial arts action synthesis method according to claim 6, characterized in that: The prior text encoder includes a CLIP pre-training model, and the CLIP pre-training model is used to encode the historical martial arts prompt text to obtain historical text conditional features; The prior action encoder includes a fifth linear layer, and the fifth linear layer is used to encode historical martial arts action clips to obtain historical action features.

8. The semantic-driven martial arts action synthesis method according to claim 5, characterized in that: The method of generating a noise action segment corresponding to the final noise scale as a martial arts action segment corresponding to the current autoregressive iteration step based on the text conditional features, the action duration, and the historical prior condition vector through reverse diffusion iteration includes: Taking text conditional features, action duration, historical prior condition vector, current noise scale, and noisy action fragments obtained at the current noise scale as input data, a single-step inverse diffusion network is used to generate rough clean action fragments based on the input data. Specifically, the second linear layer, the second MLP, and the third linear layer respectively map the noisy action fragments obtained at the current noise scale, the current noise scale, and the text conditional features. The three mapping results, the historical prior condition vector, and the result of position encoding of the action duration are then concatenated to obtain a concatenated vector. The concatenated vector is then encoded by the Transformer encoder and linearly mapped by the fourth linear layer to obtain a rough clean action fragment. Based on the rough clean action fragment, forward diffusion is performed to obtain the noise action fragment corresponding to the next noise scale, and then the noise action fragment corresponding to the final noise scale is generated through iterative reverse diffusion as the martial arts action fragment corresponding to the current autoregressive iteration step.

9. The semantic-driven martial arts action synthesis method according to claim 1, characterized in that: The method also includes constructing an action-martial arts prompt text dataset, and specifying the martial arts prompt text of the martial arts action clip to be synthesized based on the martial arts action-martial arts prompt text dataset.

10. The semantic-driven martial arts action synthesis method according to claim 1, characterized in that: It also includes visual display of martial arts action clips, including timing visualization and spatial visualization. Among them, timing visualization provides a playback progress bar to play the action sequence of martial arts action clips in timing animation and jump to specified time frames; spatial visualization will provide two modes of visualization of the three-dimensional posture of each frame, namely skeleton and grid.

Images