Motion generation method, apparatus, equipment, and storage medium based on multimodal signals
Patent Information
- Application Number
- CN202510890022.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-30
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-06-30
Smart Images

Figure CN120931775B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of motion control technology, and more specifically, to a motion generation method, apparatus, device, and storage medium based on multimodal signals. Background Technology
[0002] In recent years, multimodal conditional motion generation has attracted considerable attention. Researchers have explored combining multiple modalities, such as text and trajectories, to generate motion sequences suitable for a range of specific application scenarios, such as virtual reality and human-computer interaction. In particular, text- and trajectory-guided motion generation methods have become a research hotspot due to their strong controllability. They can generate the required fine-grained motion sequences based on user-provided text descriptions and the motion trajectories of specific human key points.
[0003] Since motion generation is guided by both text and trajectory signals, current methods have the following problems: (1) the generated motion may follow multiple semantics but deviate from trajectory control; (2) the generated motion may not match the text description and over-rely on trajectory information. Current methods often fail to balance the coordinated control between text and trajectory signals, causing the generated motion to deviate from expectations. Summary of the Invention
[0004] This application provides a motion generation method, apparatus, device, and storage medium based on multimodal signals, to at least solve the technical problem in related technologies where the inability to balance the coordinated control between text and trajectory signals leads to the generated motion deviating from the expected direction.
[0005] According to one aspect of the embodiments of this application, a motion generation method based on multimodal signals is provided, comprising:
[0006] The signal density measurement module of the motion generation model is balanced by inputting text descriptions and trajectory control signals, and outputs dense semantic signals and sparse semantic signals.
[0007] The dense semantic signal, sparse semantic signal, and trajectory control signal are input into the text-trajectory signal balancing module of the motion generation model, which outputs the context semantic modulation signal and the spatial trajectory modulation signal.
[0008] The context semantic modulation signal and the spatial trajectory modulation signal are used as conditional inputs to generate optimized motion features;
[0009] Based on the optimized motion features, an attention mechanism and a linear layer are used to denoise the noisy motion sequence to generate the target motion sequence.
[0010] In one implementation, before the signal density measurement module for balancing the text description and trajectory control signal input signals to the motion generation model, the method further includes:
[0011] The signal equilibrium motion generation model is constructed based on the motion diffusion model;
[0012] The signal balance motion generation model incorporates the signal density measurement module and the text-trajectory signal balance module.
[0013] In one implementation, the signal density measurement module of the motion generation model inputs text description and trajectory control signals to balance the signal density of the motion generation model, and outputs dense semantic signals and sparse semantic signals, including:
[0014] The input text description is segmented into words to obtain a word sequence;
[0015] For each time step, calculate the cross-attention score between each word sequence and the motion feature;
[0016] The dense semantic signal and the sparse semantic signal are determined based on the cross-attention score between the word sequence and the motion features.
[0017] In one implementation, the dense semantic signal is determined based on the cross-attention score between the word sequence and motion features, including:
[0018] Based on the cross-attention score between the word sequence and motion features, determine the minimum attention value between all words and motion features at each time step;
[0019] The current time step has dense semantic signals when the minimum attention value at the current time step is greater than or equal to the average attention value at all time steps.
[0020] In one implementation, the sparse semantic signal is determined based on the cross-attention score between the word sequence and motion features, including:
[0021] Based on the cross-attention score between the word sequence and motion features, determine the maximum attention value between all words and motion features at each time step;
[0022] The current time step has a sparse semantic signal when the maximum attention value at the current time step is less than or equal to the average attention value at all time steps.
[0023] In one implementation, the dense semantic signal, sparse semantic signal, and trajectory control signal are input into the text-trajectory signal balancing module of the motion generation model, and the output context semantic modulation signal and spatial trajectory modulation signal are included:
[0024] Dense semantic signals, sparse semantic signals, and trajectory control signals are converted into multi-scale features and fused to obtain a combined feature vector.
[0025] The combined feature vector is transformed using a linear layer with a Sigmoid activation function to generate a modulation factor;
[0026] Based on the modulation factor, the context semantic modulation signal and the spatial trajectory modulation signal are generated.
[0027] In one implementation, the context semantic modulation signal and the spatial trajectory modulation signal are used as conditional inputs to generate optimized motion features, including:
[0028] Based on contextual semantic modulation signals and spatial trajectory modulation signals, motion features are fused within the masked region;
[0029] The fused features are fine-tuned using a cross-attention mechanism to generate the optimized motion features.
[0030] According to another aspect of the embodiments of this application, a motion generation device based on multimodal signals is provided, comprising:
[0031] The density measurement module is used to balance the input signals of text description and trajectory control signals into the motion generation model, and output dense semantic signals and sparse semantic signals.
[0032] The text-trajectory balancing module is used to balance the dense semantic signal, sparse semantic signal and trajectory control signal input signals into the motion generation model's text-trajectory signal balancing module, and outputs context semantic modulation signal and spatial trajectory modulation signal.
[0033] The optimization module is used to take the context semantic modulation signal and the spatial trajectory modulation signal as conditional inputs to generate optimized motion features;
[0034] The denoising module is used to denoise the noisy motion sequence based on the optimized motion features, using an attention mechanism and a linear layer, to generate the target motion sequence.
[0035] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-described motion generation method based on multimodal signals through the computer program.
[0036] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, and the computer program is configured to execute the above-described motion generation method based on multimodal signals when running.
[0037] The technical solutions provided in this application embodiment may include the following beneficial effects:
[0038] This application proposes a signal-balanced motion generation model to address the challenge of coordinating text and trajectory signals in multimodal motion generation. A signal density measurement module and a text-trajectory signal balancing module are introduced. This application conducts an in-depth analysis of the coordination requirements of text and trajectory signals in motion control, achieving precise control through semantic information density analysis. A text-trajectory signal balancing structure is developed, effectively mitigating the negative impact of signal conflicts on motion generation. It effectively solves the problem of generated motion sequences not meeting user expectations. Attached Figure Description
[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0040] Figure 1 This is a flowchart of a motion generation method based on multimodal signals according to an embodiment of this application;
[0041] Figure 2 This is a schematic diagram of an existing motion generation method;
[0042] Figure 3 This is a schematic diagram of a signal balance motion generation model according to an embodiment of this application;
[0043] Figure 4 This is a schematic diagram of a signal density measurement module according to an embodiment of this application;
[0044] Figure 5 This is a schematic diagram of a text-track signal balancing module according to an embodiment of this application;
[0045] Figure 6 This is a schematic diagram of a motion generation device based on multimodal signals according to an embodiment of this application;
[0046] Figure 7 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation
[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0048] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0049] like Figure 2 As shown, motion generation is guided by both text and trajectory signals. Problems with existing methods include deviations from the expected trajectory and mismatch with the text, resulting in unbalanced signals.
[0050] Based on this, this application proposes a signal equilibrium motion generation model (SBMG), such as Figure 3 As shown, the system includes a Signal Density Measurement (SDM) module and a Text-Trajectory Signal Balancing (TSB) module. The input text description and trajectory are first processed by the SDM module to generate dense and sparse semantic signals. These signals are then passed to the TSB module, which obtains a balanced embedding representation between the semantic and trajectory signals at different time points. Based on dynamically adjusting intermediate motion features F, it generates optimized motion features F′. The optimized features are then input to an attention module, which, based on an attention mechanism and a linear layer, processes the noisy motion sequence X. t Denoising is performed to generate the target motion sequence X0. This enables precise control over the motion sequence.
[0051] The following is in conjunction with the appendix Figure 1 The motion generation method based on multimodal signals according to embodiments of this application will be described in detail. For example... Figure 1 As shown, the method mainly includes the following steps:
[0052] S101 inputs text description and trajectory control signals to the signal density measurement module of the motion generation model, and outputs dense semantic signals and sparse semantic signals.
[0053] In one implementation, before inputting the text description and trajectory control signals into the signal density measurement module of the signal balancing motion generation model, the method further includes: constructing a signal balancing motion generation model based on a motion diffusion model, wherein the signal balancing motion generation model is introduced into the signal density measurement module and the text-trajectory signal balancing module.
[0054] This application proposes a simple yet effective framework that dynamically adjusts the influence of text and trajectory control signals to generate more natural, coherent motion sequences that meet user expectations. This method is based on the Motion Diffusion Model (MDM) and introduces two key modules: a signal density measurement module and a text-trajectory signal balancing module.
[0055] Motion diffusion model (MDM) is a machine learning model for generating motion sequences, drawing inspiration from diffusion models. Originally used for image generation, diffusion models gradually transform noisy data into target data by simulating the diffusion process in the physical world. MDM applies this concept to motion data generation, simulating the diffusion process of motion data to gradually generate motion sequences that meet specific conditions. In MDM, motion sequence generation is modeled as a progressive denoising process. The model starts with a simple noisy motion sequence and then gradually adds details until a clear motion sequence that meets specific conditions is generated.
[0056] First, the signal density measurement module of the motion generation model is used to input text descriptions and trajectory control signals to balance the motion generation model, and outputs dense semantic signals and sparse semantic signals.
[0057] In actual interaction, users can input a text description of the desired motion and specify the trajectory of key body points. The model uses these multimodal control signals as input to generate motion sequences.
[0058] Existing multimodal motion generation methods lack effective mechanisms to measure and identify overly dense or sparse semantic signals, which affects the accuracy of generated motion. The Signal Density Measurement (SDM) module addresses this problem by analyzing the alignment between different semantic signals and motion features, extracting the information density distribution across different time points, and providing precise adjustment signals for subsequent generation processes.
[0059] Specifically, the input text description is segmented into words to obtain a word sequence; for each time step, the cross-attention score between each word sequence and the motion feature is calculated; based on the cross-attention score between the word sequence and the motion feature, dense semantic signals and sparse semantic signals are determined.
[0060] The input text description is first segmented to obtain a vocabulary sequence {vocabulary1, vocabulary2, ..., vocabularyn}. Then, for each time step t, the cross-attention score s between each lexical feature and the motion feature is calculated as follows:
[0061] s = E T ·X T
[0062] Among them, E T Indicates the encoded text embedding, x T The alignment score S represents the degree of matching between lexical features and motion features at each time step.
[0063] To detect dense and sparse semantic cases, the information density distribution of each signal at different time points is analyzed. From S, the minimum and maximum values at each time step are extracted, and mean filtering is applied to preserve important signal information.
[0064] In one implementation, the minimum attention value of all words and motion features at each time step is determined based on the cross-attention score between the word sequence and motion features; when the minimum attention value at the current time step is greater than or equal to the average attention value at all time steps, the current time step has a dense semantic signal.
[0065] If, at a certain time step, the attention distribution of lexical features to action features is highly dense, that is, at this time step, the minimum value of the attention of all lexical and action features is greater than the mean value, then this time step has a semantically dense signal.
[0066] In one implementation, determining a sparse semantic signal based on the cross-attention score between the word sequence and motion features includes: determining the maximum attention value of all words and motion features at each time step based on the cross-attention score between the word sequence and motion features; and determining that the current time step has a sparse semantic signal when the maximum attention value at the current time step is less than or equal to the average attention value at all time steps.
[0067] When the attention distribution of lexical features to motion features is sparse at a certain time step, that time step is labeled as a sparse semantic signal.
[0068] like Figure 4 As shown, the SDM module details are presented. This module accepts text and motion trajectory inputs, segments the input text description into words to obtain a word sequence; for each time step, it calculates the cross-attention score between each word sequence and motion features; based on the cross-attention score between the word sequence and motion features, it generates dense semantic signals and sparse semantic signals for dynamic adjustment in subsequent motion generation.
[0069] By employing both dense and sparse semantic signals, the SDM (Signal Density Measurement Module) effectively provides adjustment signals for motion generation. These signals are then processed in the TSB (Text-Trajectory Signal Balancing Module) to generate more accurate dynamic modulation signals, improving the coordinated control between text and trajectory.
[0070] S102 is the text-trajectory signal balancing module of the motion generation model that balances dense semantic signals, sparse semantic signals, and trajectory control signals as input signals, and outputs context semantic modulation signals and spatial trajectory modulation signals.
[0071] Although the signal density measurement module effectively identifies dense and sparse semantic signals, dynamically adjusting the weights of text and trajectory signals during generation to achieve precise semantic and trajectory control remains a challenge. The main task of the TSB (Text-Trajectory Signal Balancing Module) is to generate balanced embedding representations at different time steps based on the signals from the SDM (Signal Density Measurement Module), thereby achieving dynamic adjustment of text and trajectory signals.
[0072] By generating contextual semantic modulation signals and spatial trajectory modulation signals, the model coordinates the guiding influence of each signal on the generated motion, enhancing the coherence and accuracy of the sequence.
[0073] In one implementation, dense semantic signals, sparse semantic signals, and trajectory control signals are converted into multi-scale features and fused to obtain a combined feature vector. The combined feature vector is then transformed using a linear layer with a sigmoid activation function to generate a modulation factor; based on the modulation factor, the generation of context semantic modulation signals and spatial trajectory modulation signals is controlled.
[0074] Specifically, in the text-trajectory signal balancing module, dense semantic signals (sig_dense), sparse semantic signals (sig_sparse), and trajectory sequence signals (x_seq) are initially abstracted into multi-scale features and fused. These features are then concatenated into a combined feature vector. This combined vector is transformed through a linear layer with a Sigmoid activation function, producing a modulation factor ρ.
[0075] ρ = σ(W·combined + b);
[0076] Here, σ represents the Sigmoid activation function, and W and b are the weight matrix and bias term of the linear transformation, respectively. The modulation factor ρ controls the generation of the context semantic modulation signal and the spatial trajectory modulation signal. The modulation factor is a value between 0 and 1, calculated from a linear layer in the model and the Sigmoid activation function. This factor controls the weights of two important signals during generation. For example, if the model determines that it relies more on the text description to generate motion, the value of the modulation factor may be more inclined to enhance the influence of the context semantic modulation signal; conversely, if the model determines that it needs to more strictly follow the trajectory signal, the value of the modulation factor may be more inclined to enhance the influence of the spatial trajectory modulation signal.
[0077] like Figure 5As shown, the TSB module receives dense semantic signals, sparse semantic signals, and motion trajectory sequences, converts them into multi-scale features, and performs fusion processing to obtain a combined feature vector. A linear layer with a sigmoid activation function is used to transform the combined feature vector, generating a modulation factor. Based on the modulation factor, the generation of context semantic modulation signals and spatial trajectory modulation signals is controlled. Further, based on the context semantic modulation signals and spatial trajectory modulation signals, they are fused with motion features F within a masked region. A cross-attention mechanism is used to fine-tune the fused features, generating optimized motion features F′.
[0078] This application can generate balanced embedding representations at different time steps, thereby enabling dynamic adjustment of text and trajectory signals.
[0079] S103 uses the context semantic modulation signal and the spatial trajectory modulation signal as conditional inputs to generate optimized motion features.
[0080] In one implementation, the context semantic modulation signal and the spatial trajectory modulation signal are used as conditional inputs to generate optimized motion features, including:
[0081] Based on contextual semantic modulation signals and spatial trajectory modulation signals, motion features F are fused within the masked region; the fused features are then fine-tuned using a cross-attention mechanism to generate optimized motion features F′.
[0082] Specifically, contextual semantic modulation signals and spatial trajectory modulation signals are used as conditional inputs, which are then fused with motion features F within a defined masked region. The fused features are further adjusted for the influence of text and trajectory signals through a cross-attention mechanism, resulting in dynamically optimized motion features F′. This balancing process significantly enhances the model's ability to handle both dense and sparse semantic signals, ensuring that the generated motion sequences better meet user expectations.
[0083] S104 uses optimized motion features, an attention mechanism, and a linear layer to denoise noisy motion sequences and generate target motion sequences.
[0084] In one implementation, optimized motion features are input into an attention module, and further, based on the attention mechanism and a linear layer, the noisy motion sequence X is processed. t Denoise the target motion sequence X0.
[0085] Inputting optimized motion features into the attention module can further enhance the model's ability to capture key information. The attention mechanism can provide an explanation of the model's decision-making process. By analyzing the attention weights, we can understand which features the model focuses on when generating motion, thereby enhancing the model's interpretability. Moreover, inputting optimized features into the attention module can provide higher-quality input for subsequent denoising steps. This helps improve denoising results and generate smoother motion sequences that conform to physical laws.
[0086] Furthermore, this application defines a loss function by calculating the masked mean squared error (MSE) between the model output and the target within the masked region, thereby improving the model's accuracy in multimodal control tasks. The loss function is expressed as follows:
[0087]
[0088] Where, m i It is a mask value, indicating the effective region for loss calculation; t i It is the target, representing the expected reference output; i is the model output, representing the prediction result generated by the model; N is the batch size or number of samples; J represents the number of joints, describing the composition of motion features; D is the feature dimension of each joint, describing the fine-grained representation of motion features. This loss function design focuses on the effective region through a masking mechanism, ensuring generation quality and control accuracy in the event of multimodal signal conflict.
[0089] This application proposes a Signal Balanced Motion Generator (SBMG) to address the challenge of coordinating text and trajectory signals in multimodal motion generation. The SBMG framework comprises a signal density measurement module and a text-trajectory signal balancing module, which effectively solve the problem of generated motion sequences not meeting user expectations. First, the coordination requirements of text and trajectory signals in motion control are analyzed in depth, and precise control is achieved through semantic information density analysis. Second, a text-trajectory signal balancing structure is developed to effectively mitigate the negative impact of signal conflicts on motion generation. Furthermore, the SBMG framework's universality and efficiency are validated on multiple datasets, demonstrating significant improvements in generation quality and control accuracy. These results not only provide new directions for future research in multimodal motion generation but also offer important technical support for applications in related fields. In summary, SBMG significantly improves the naturalness and accuracy of motion generation by balancing the influence of text and trajectory signals, and has broad application prospects.
[0090] According to another aspect of the embodiments of this application, a motion generation apparatus based on multimodal signals is also provided for implementing the above-described motion generation method based on multimodal signals. For example... Figure 6 As shown, the device includes:
[0091] The density measurement module 601 is used to input text description and trajectory control signal into the signal density measurement module of the motion generation model, and output dense semantic signal and sparse semantic signal.
[0092] The text-trajectory balancing module 602 is used to balance the input signals of dense semantic signals, sparse semantic signals and trajectory control signals in the motion generation model, and outputs context semantic modulation signals and spatial trajectory modulation signals.
[0093] The optimization module 603 is used to take the context semantic modulation signal and the spatial trajectory modulation signal as conditional inputs to generate optimized motion features;
[0094] The denoising module 604 is used to denoise the noisy motion sequence based on optimized motion features, using an attention mechanism and a linear layer to generate the target motion sequence.
[0095] It should be noted that the motion generation device based on multimodal signals provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the motion generation method based on multimodal signals. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the motion generation device based on multimodal signals provided in the above embodiments and the motion generation method based on multimodal signals are based on the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0096] According to another aspect of the embodiments of this application, an electronic device corresponding to the motion generation method based on multimodal signals provided in the foregoing embodiments is also provided, so as to execute the motion generation method based on multimodal signals described above.
[0097] Please refer to Figure 7 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device includes: a processor 700, a memory 701, a bus 702, and a communication interface 703. The processor 700, the communication interface 703, and the memory 701 are connected via the bus 702. The memory 701 stores a computer program that can run on the processor 700. When the processor 700 runs the computer program, it executes the motion generation method based on multimodal signals provided in any of the foregoing embodiments of this application.
[0098] The memory 701 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 703 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.
[0099] Bus 702 can be an ISA bus, PCI bus, or EISA bus, etc. Buses can be divided into address buses, data buses, control buses, etc. Memory 701 is used to store programs. After receiving execution instructions, processor 700 executes the program. The motion generation method based on multimodal signals disclosed in any of the foregoing embodiments of this application can be applied to processor 700, or implemented by processor 700.
[0100] The processor 700 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 700 or by instructions in software form. The processor 700 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 701. Processor 700 reads the information in memory 701 and, in conjunction with its hardware, completes the steps of the above method.
[0101] The electronic device provided in this application embodiment and the motion generation method based on multimodal signals provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.
[0102] According to another aspect of the embodiments of this application, a computer-readable storage medium corresponding to the motion generation method based on multimodal signals provided in the foregoing embodiments is also provided, wherein a computer program (i.e., a program product) is stored thereon, and when the computer program is run by a processor, it executes the motion generation method based on multimodal signals provided in any of the foregoing embodiments.
[0103] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.
[0104] The computer-readable storage medium provided in the above embodiments of this application and the motion generation method based on multimodal signals provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.
[0105] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0106] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.
Claims
1. A motion generation method based on multimodal signals, characterized in that, include: The signal density measurement module of the motion generation model inputs text description and trajectory control signals to balance the signal density measurement module, and outputs dense semantic signals and sparse semantic signals. This includes: segmenting the input text description to obtain a word sequence; calculating the cross-attention score between each word sequence and motion features for each time step; determining the minimum attention value between all words and motion features at each time step based on the cross-attention score; indicating that the current time step has a dense semantic signal when the minimum attention value at the current time step is greater than or equal to the average attention value at all time steps; determining the maximum attention value between all words and motion features at each time step based on the cross-attention score; indicating that the current time step has a sparse semantic signal when the maximum attention value at the current time step is less than or equal to the average attention value at all time steps. The dense semantic signal, sparse semantic signal, and trajectory control signal are input into the text-trajectory signal balancing module of the motion generation model, which outputs the context semantic modulation signal and the spatial trajectory modulation signal. The context semantic modulation signal and the spatial trajectory modulation signal are used as conditional inputs to generate optimized motion features; Based on the optimized motion features, an attention mechanism and a linear layer are used to denoise the noisy motion sequence to generate the target motion sequence.
2. The method according to claim 1, characterized in that, Before balancing the signal density measurement module of the motion generation model with the text description and trajectory control signal input signals, it also includes: The signal equilibrium motion generation model is constructed based on the motion diffusion model; The signal balance motion generation model incorporates the signal density measurement module and the text-trajectory signal balance module.
3. The method according to claim 1, characterized in that, The text-trajectory signal balancing module of the motion generation model inputs the dense semantic signal, sparse semantic signal, and trajectory control signal to output context semantic modulation signal and spatial trajectory modulation signal, including: Dense semantic signals, sparse semantic signals, and trajectory control signals are converted into multi-scale features and fused to obtain a combined feature vector. The combined feature vector is transformed using a linear layer with a Sigmoid activation function to generate a modulation factor; Based on the modulation factor, the context semantic modulation signal and the spatial trajectory modulation signal are generated.
4. The method according to claim 1, characterized in that, Using the context semantic modulation signal and the spatial trajectory modulation signal as conditional inputs, optimized motion features are generated, including: Based on contextual semantic modulation signals and spatial trajectory modulation signals, motion features are fused within the masked region; The fused features are fine-tuned using a cross-attention mechanism to generate the optimized motion features.
5. A motion generation device based on multimodal signals, characterized in that, include: A density measurement module is used to input text descriptions and trajectory control signals into a signal density measurement module of a motion generation model, and output dense semantic signals and sparse semantic signals. This includes: segmenting the input text description to obtain a word sequence; calculating the cross-attention score between each word sequence and the motion feature for each time step; determining the minimum attention value between all words and motion features at each time step based on the cross-attention score; indicating that the current time step has a dense semantic signal when the minimum attention value at the current time step is greater than or equal to the average attention value at all time steps; determining the maximum attention value between all words and motion features at each time step based on the cross-attention score; indicating that the current time step has a sparse semantic signal when the maximum attention value at the current time step is less than or equal to the average attention value at all time steps. The text-trajectory balancing module is used to balance the dense semantic signal, sparse semantic signal and trajectory control signal input signals into the motion generation model's text-trajectory signal balancing module, and outputs context semantic modulation signal and spatial trajectory modulation signal. The optimization module is used to take the context semantic modulation signal and the spatial trajectory modulation signal as conditional inputs to generate optimized motion features; The denoising module is used to denoise the noisy motion sequence based on the optimized motion features, using an attention mechanism and a linear layer, to generate the target motion sequence.
6. An electronic device, characterized in that, It includes a processor and a memory storing program instructions, the processor being configured to, when executing the program instructions, perform the motion generation method based on multimodal signals as described in any one of claims 1 to 4.
7. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement a motion generation method based on multimodal signals as described in any one of claims 1 to 4.
