Motion synthesis method and device and electronic equipment
By superimposing Gaussian noise onto keypoint sequences using a diffusion probability model and combining it with semantic feature guidance, the robustness and diversity issues of human motion skeleton sequence synthesis algorithms are solved, generating actions that conform to physical laws and semantic consistency, applicable to fields such as virtual human synthesis and film and animation.
Patent Information
- Application Number
- CN202510849729.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-11-18
AI Technical Summary
Existing algorithms for synthesizing human motion skeleton sequences are not robust, lack richness and diversity in motion generation, and produce the same inference results each time.
Gaussian noise is superimposed on the keypoint sequence using a diffusion probability model. Combined with semantic feature guidance, the final skeleton sequence is gradually denoised. The keypoint sequence is then denoised using a diffusion probability model, and prediction and denoising are performed under the guidance of semantic features to generate a skeleton sequence that conforms to physical laws and is semantically consistent.
It improves the robustness and diversity of human motion synthesis, and the generated motions conform to physical laws and are consistent with semantics, providing a more natural and controllable motion generation scheme.
Smart Images
Figure CN120976378A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more specifically, to an action synthesis method, apparatus, and electronic device. Background Technology
[0002] Body posture and facial expressions are crucial ways of conveying information in human communication, and are closely related to the key technologies for developing realistic avatars in movies, games, virtual social interactions, and interactions with social robots. Human motion generation aims to generate natural human posture sequences and has significant potential for practical applications. In recent years, advancements in motion data collection technologies and generation methods have laid the foundation for the development of this field, especially in the field of text-based video, where related technologies (such as Sora) have attracted attention. In the synthesis of virtual humans in many scenarios, matching characters are often generated based on skeletal sequences and other conditions.
[0003] However, current human motion generation methods suffer from numerous problems. Current algorithms for synthesizing human motion skeleton sequences are conditional, relying on factors such as text, audio, and scene. Most existing motion generation schemes are trained on specific individuals, leading to poor robustness of the generated motions, limited richness and diversity, and identical results in each inference iteration. Summary of the Invention
[0004] To address the problems existing in the prior art, embodiments of this application provide an action synthesis method, apparatus, and electronic device. By introducing a diffusion probability model and embedding conditions such as speech and semantics, the robustness of human action synthesis can be improved.
[0005] In a first aspect, embodiments of this application provide an action synthesis method, including:
[0006] The video sequence is detected to obtain a key point sequence, which is then visualized and aligned with the human skeleton to obtain a preliminary skeleton sequence.
[0007] Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence over time, with longer time intervals receiving more Gaussian noise; and
[0008] Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
[0009] Furthermore, after detecting keypoint sequences in the video sequence and performing visualization processing to align them with a human skeleton to obtain a preliminary skeleton sequence, the method further includes:
[0010] The preliminary skeleton sequence is encoded.
[0011] Furthermore, after employing a diffusion probability model to superimpose different levels of Gaussian noise onto the keypoint sequence over time, wherein the longer the time interval, the more Gaussian noise is added, the process further includes:
[0012] The initial skeleton sequence after noise addition is mapped to the latent vector space of the diffusion probability model through positional encoding and a linear layer; and
[0013] Time is encoded as the number of sampling steps and mapped to the latent vector space of the diffusion probability model through a linear layer.
[0014] Furthermore, after encoding time as the number of sampling steps and mapping it to the latent vector space of the diffusion probability model through a linear layer, the method further includes:
[0015] If the time exceeds the preset time, a distributed guidance method will be used for motion synthesis.
[0016] Furthermore, after the initial skeleton sequence encoded with noise is mapped to the latent vector space of the diffusion probability model through positional encoding and a linear layer, the following steps are also included:
[0017] The scale and style features of the characters in the video are learned and encoded, mapped to the latent vector space of the diffusion probability model through a linear layer, and concatenated with the features of the initial skeleton sequence, and then mapped to the latent vector space of the diffusion probability model through another linear layer.
[0018] Further, the step of using the noisy keypoint sequence as a condition, guided by semantic features, employing the diffusion probability model to predict noise, and obtaining the final skeleton sequence after denoising includes:
[0019] The noisy keypoint sequence is used as a condition, and a semantic vector representing semantic features is superimposed. The diffusion probability model is used to predict the noise, and the final skeleton sequence is obtained after denoising.
[0020] Furthermore, before obtaining the final skeleton sequence after denoising by using the noisy keypoint sequence as a condition and guiding it with semantic features, the method further includes:
[0021] Separate audio from video and extract audio features; and
[0022] Rhythm-prosodic features and MFCC spectral features are introduced and concatenated with the audio features, and then mapped to the latent vector space of the diffusion probability model through a linear layer.
[0023] Secondly, embodiments of this application also provide an action synthesis device, comprising:
[0024] The preliminary skeleton sequence acquisition module is used to detect key point sequences in video sequences, perform visualization processing, and align them with human skeletons to obtain preliminary skeleton sequences.
[0025] The noise addition module is used to apply Gaussian noise of varying degrees to the keypoint sequence over time using a diffusion probability model, wherein the longer the time interval, the more Gaussian noise is added; and
[0026] The final skeleton sequence generation module is used to take the noisy keypoint sequence as a condition, and under the guidance of semantic features, use the diffusion probability model to predict noise, and obtain the final skeleton sequence after denoising.
[0027] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the program to implement the action synthesis method according to the first aspect described above.
[0028] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, the computer program being used to implement the action synthesis method according to the first aspect described above.
[0029] Fifthly, embodiments of this application also provide a computer program product having a computer program stored thereon, the computer program being used to implement the action synthesis method according to the first aspect described above.
[0030] The embodiments of this application bring the following beneficial effects:
[0031] In the motion synthesis method provided in this application embodiment, firstly, keypoint detection is performed on the input video sequence to extract human posture keypoint information. After visualization processing and skeleton alignment, a preliminary skeleton sequence is obtained, laying the foundation for subsequent processing. Secondly, a diffusion probability model is used to add noise to the keypoint sequence, superimposing different levels of Gaussian noise over time. The longer the time, the more noise is added. This strategy can better simulate dynamic changes in motion, capture the temporal dependence of motion, and enable the model to have a more detailed perception of motion changes. Finally, using the noisy keypoint sequence as a condition, under the guidance of semantic features, the diffusion probability model is used to predict and remove noise to generate the final skeleton sequence. Semantic feature guidance covers high-level information such as action category and emotional state, ensuring that the generated action matches specific semantics and improving the rationality and diversity of the action. Through gradual denoising, the model recovers a clear skeleton sequence, ensuring that the generated action conforms to physical laws and is consistent with semantics. The motion synthesis method provided in this application embodiment effectively combines keypoint detection, diffusion probability model, and semantic guidance technology, which can solve the problems of robustness, long sequence fluency, and action diversity in traditional motion generation, providing a more natural and controllable motion generation solution for fields such as virtual human synthesis and film and animation. Attached Figure Description
[0032] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0033] Figure 1 A flowchart illustrating the action synthesis method provided in this application embodiment;
[0034] Figure 2 This is a schematic diagram of the actual operation flow of the action synthesis method provided in the embodiments of this application;
[0035] Figure 3 A schematic flowchart illustrating the long-term continuous processing in the action synthesis method provided in this application embodiment;
[0036] Figure 4 A schematic diagram of the diffusion probability model architecture used in the action synthesis method provided in the embodiments of this application;
[0037] Figure 5 This is a structural block diagram of the motion synthesis device provided in the embodiments of this application;
[0038] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0039] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of this application.
[0041] In the specification, claims, and accompanying drawings of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.
[0042] Figure 1 and Figure 2 These are a flowchart and a schematic diagram of the actual operation flow of an action synthesis method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the action synthesis method of this application embodiment includes the following steps:
[0043] S101: Detect keypoint sequences in the video sequence, perform visualization processing and align them with the human skeleton to obtain a preliminary skeleton sequence;
[0044] S102: Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence according to time, with more Gaussian noise added for longer time intervals; and
[0045] S103: Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
[0046] Specifically, in the action synthesis method provided in this application embodiment, firstly, keypoint detection is performed on the input video sequence to extract human posture keypoint information. After visualization processing and alignment with the skeleton, a preliminary skeleton sequence is obtained, laying the foundation for subsequent processing. Secondly, a diffusion probability model is used to add noise to the keypoint sequence, superimposing Gaussian noise of different degrees over time. The longer the time, the more noise is added. This strategy can better simulate dynamic changes in actions, capture the temporal dependence of actions, and enable the model to have a more detailed perception of action changes. Finally, using the noisy keypoint sequence as a condition, under the guidance of semantic features, the diffusion probability model is used to predict and remove noise to generate the final skeleton sequence. The semantic feature guidance covers high-level information such as action category and emotional state, ensuring that the generated action matches specific semantics and improving the rationality and diversity of actions. Through gradual denoising, the model recovers a clear skeleton sequence, ensuring that the generated action conforms to physical laws and is consistent with semantics. The motion synthesis method provided in this application effectively combines key point detection, diffusion probability model and semantic guidance technology, which can solve the problems of robustness, long sequence fluency and motion diversity in traditional motion generation, and provide a more natural and controllable motion generation solution for fields such as virtual human synthesis and film and animation.
[0047] Furthermore, after detecting keypoint sequences in the video sequence and performing visualization processing to align them with a human skeleton to obtain a preliminary skeleton sequence, the method further includes:
[0048] The preliminary skeleton sequence is encoded.
[0049] Specifically, in the motion synthesis method provided in this application embodiment, the input video sequence is first detected to accurately obtain a keypoint sequence. Then, visualization processing and alignment with a human skeleton are performed to obtain a preliminary skeleton sequence, which provides basic structural information for subsequent motion generation. Based on this, the preliminary skeleton sequence needs to be encoded. The encoding process may convert the skeleton sequence into a specific format or feature representation to facilitate subsequent model processing. This process helps extract key features from the skeleton sequence, compress data dimensionality, and retain important motion information, providing a more efficient and easier-to-process input for subsequent operations using a diffusion probability model.
[0050] Furthermore, after employing a diffusion probability model to superimpose different levels of Gaussian noise onto the keypoint sequence over time, wherein the longer the time interval, the more Gaussian noise is added, the process further includes:
[0051] The initial skeleton sequence after noise addition is mapped to the latent vector space of the diffusion probability model through positional encoding and a linear layer; and
[0052] Time is encoded as the number of sampling steps and mapped to the latent vector space of the diffusion probability model through a linear layer.
[0053] Specifically, the initial skeleton sequence after noise processing is positionally encoded, and then mapped to the latent vector space of the diffusion probability model through a linear layer. Positional encoding assigns unique identifiers to key points at different positions in the sequence, enabling the model to better understand the temporal structure of the sequence; the linear layer mapping transforms the data into a latent space suitable for model processing, facilitating subsequent operations.
[0054] Simultaneously, time is encoded as the number of sampling steps, and then mapped to the latent vector space of the diffusion probability model through a linear layer. Temporal encoding allows the model to clearly define the position of the action on the time axis, which helps to capture the pattern of action evolution over time. For example, time t is an integer between 0 and 1000, representing the number of sampling steps. This time is mapped to a 256-dimensional feature X_ti∈
[256] through a linear layer, and then repeated n times on the frame to obtain X_ti∈[n, 256].
[0055] Furthermore, after encoding time as the number of sampling steps and mapping it to the latent vector space of the diffusion probability model through a linear layer, the method further includes:
[0056] If the time exceeds the preset time, a distributed guidance method will be used for motion synthesis.
[0057] Specifically, if the time exceeds the preset time, a distributed guidance method is used for motion synthesis. Based on specific motion distribution patterns or prior knowledge, distributed guidance can constrain and guide motion generation according to the distribution characteristics of preceding frames when generating long sequences, ensuring that the generated motion remains reasonable and coherent over a long period of time, and avoiding motion distortion or incoordination due to excessively long sequences.
[0058] For example, refer to Figure 3 In some embodiments of this application, a training method based on keypoint sequence guidance is proposed for the synthesis of long-duration action sequences. The entire process can be as follows: Figure 3 As shown. Taking each inference as an example of 220 frames, the first time, since there is no guide frame, 220 frames of random Gaussian noise are input to generate 220 frames of results. The second time, the last 20 frames of the first inference are used as guide frames, and concatenated with 200 frames of random noise in the time dimension, then the 220 frames of results are output, only the last 200 frames are taken, and so on. Due to the above characteristics of inference, there are two ways to train the diffusion probability model: Method 1 is an experiment that uses 20 frames of ground truth to guide the generation of 200 frames of results; Method 2 is an experiment that uses 220 frames of random noise for denoising.
[0059] Furthermore, after the initial skeleton sequence encoded with noise is mapped to the latent vector space of the diffusion probability model through positional encoding and a linear layer, the following steps are also included:
[0060] The scale and style features of the characters in the video are learned and encoded, mapped to the latent vector space of the diffusion probability model through a linear layer, and concatenated with the features of the initial skeleton sequence, and then mapped to the latent vector space of the diffusion probability model through another linear layer.
[0061] Specifically, the model learns and encodes the scale and style features of the characters in the video. Scale features encompass physical characteristics such as height and body shape, while style features involve unique expressions such as movement amplitude, rhythm, and habits. After encoding these features, they are mapped to a latent vector space through a linear layer and then concatenated with features from the initial skeleton sequence. This concatenation operation integrates skeletal structure information with the character's personalized features, enabling the model to more comprehensively understand the various elements required for motion generation. The concatenated features are then mapped back to the latent vector space through a linear layer to further optimize the feature representation.
[0062] Furthermore, existing steps such as temporal encoding (as a mapping of sampling step count) also play a collaborative role. Finally, guided by semantic features, a diffusion probability model is used to predict and denoise the comprehensive features mapped to the latent space, generating the final skeleton sequence. This method significantly improves the personalization and realism of motion synthesis by fusing multi-dimensional features, generating actions that better match the characteristics of video characters, and providing a higher-quality motion generation solution for fields such as virtual human synthesis and film and animation.
[0063] Further, the step of using the noisy keypoint sequence as a condition, guided by semantic features, employing the diffusion probability model to predict noise, and obtaining the final skeleton sequence after denoising includes:
[0064] The noisy keypoint sequence is used as a condition, and a semantic vector representing semantic features is superimposed. The diffusion probability model is used to predict the noise, and the final skeleton sequence is obtained after denoising.
[0065] Specifically, the noisy keypoint sequence is used as a conditional input to the diffusion probability model, while a semantic vector representing semantic features is superimposed. The semantic vector can cover high-level semantic information such as action categories (e.g., walking, running, jumping) and emotional states (e.g., joy, sadness), providing clear directional guidance for action generation.
[0066] The diffusion probability model, based on this condition and the combination of semantic vectors, predicts noise and performs denoising operations. By progressively removing noise, the model recovers a clear and natural human skeleton sequence from the noisy sequence. In this process, the semantic vectors ensure that the generated actions are not only physically reasonable but also semantically consistent with expectations, improving the rationality and diversity of the actions.
[0067] For example, in some embodiments of this application, a semantic vector representation scheme is proposed to increase controllability and the synthesis of distinctive actions. Various common actions are labeled in the training dataset. Then, during the action labeling process, the start and end times of the corresponding actions in the video frames are labeled. The action feature is represented as X_d∈[n,10], encoded using one-hot encoding. Based on the start and end times of the action, the concept of temporal position encoding X_t∈[n,256] is introduced: for example, in a skeleton sequence of 100 frames, if the action is a greeting from frame 20 to frame 80, then X_t[20:80] is encoded using sine and cosine. Finally, the semantic feature X_d is mapped to a 256-dimensional latent space through position encoding and a linear layer, and then added to the 256-dimensional position encoding.
[0068] Furthermore, before obtaining the final skeleton sequence after denoising by using the noisy keypoint sequence as a condition and guiding it with semantic features, the method further includes:
[0069] Separate audio from video and extract audio features; and
[0070] Rhythm-prosodic features and MFCC spectral features are introduced and concatenated with the audio features, and then mapped to the latent vector space of the diffusion probability model through a linear layer.
[0071] Specifically, before the denoising generation stage, audio feature processing is required. This involves first separating the audio from the video and extracting basic audio features, then introducing rhythmic and prosodic features (such as beat and rhythmic patterns) and MFCC (Mel-Frequency Cepstral Coefficients) spectral features. These features reflect the rhythmic changes and spectral characteristics of the audio, respectively. After concatenating these features with the basic audio features, the result is mapped to the latent vector space of the diffusion probability model via a linear layer, representing them in the same space as the skeleton-related features for easier model integration. The core denoising generation stage then proceeds, using the noisy keypoint sequence as a condition, superimposing semantic vectors obtained through one-hot encoding, and combining this with the audio features mapped to the latent vector space. The diffusion probability model is then used to predict and denoise the noise, ultimately yielding the final skeleton sequence.
[0072] The motion synthesis method provided in this application introduces and integrates audio features, especially rhythm and MFCC spectral features, so that the generated motion is not only semantically reasonable, but also better matched with video audio, enhancing the rhythm and expressiveness of the motion, and providing a higher quality and more expressive motion generation solution for fields such as virtual human synthesis and film and animation.
[0073] Reference Figure 4The diagram illustrates the diffusion probability model architecture used in the action synthesis method provided in this application. The diffusion probability model is based on DIT blocks and is divided into two stages. In the first stage, audio conditions are given and embedded in the feature dimension and time dimension. ID embedding is concatenated in the feature dimension. Then, the features are transformed and unified into the latent space through a linear layer. This process is represented as MLP(concat(X_t,X_m,X_ti,X_id)∈[n,1024])∈[n,256]. Then, the features of the latent space are used as conditions and fed into four DIT blocks. The DIT blocks are constructed using the classic self-attention mechanism. In order to ensure that the ID information remains unchanged, the ID embedding is embedded in the DIT block as a condition using a cross-attention mechanism. The output feature X_o1 of the last DIT block is output through a head (output head) composed of several MLP layers and the loss function is calculated.
[0074] The second stage continues to introduce semantic conditions on the basis of the first stage latent data. After passing through a linear layer, it is also encoded into 256-dimensional features. This process is represented as MLP(concat(X_o1,X_d,X_ti,X_id)∈[n,1024])∈[n,256]: It also passes through 4 DIT blocks. In order not to lose the diversity of the first stage audio, the output features of each block of the first stage are embedded into the second stage block as conditions in the form of cross attention. The two stages are jointly trained.
[0075] Figure 5 This is a structural block diagram of the motion synthesis device 200 according to an embodiment of this application. Figure 5 As shown, the motion synthesis apparatus 200 of this application embodiment includes: a preliminary skeleton sequence acquisition module 210, a noise addition module 220, and a final skeleton sequence generation module 230, wherein:
[0076] The preliminary skeleton sequence acquisition module 210 is used to detect key point sequences in a video sequence, and perform visualization processing to align them with a human skeleton to obtain a preliminary skeleton sequence.
[0077] The noise addition module 220 is used to apply Gaussian noise of varying degrees to the keypoint sequence according to time using a diffusion probability model, wherein the longer the time, the more Gaussian noise is added; and
[0078] The final skeleton sequence generation module 230 is used to take the noisy keypoint sequence as a condition, and under the guidance of semantic features, use the diffusion probability model to predict noise, and obtain the final skeleton sequence after denoising.
[0079] In the motion synthesis apparatus provided in this application embodiment, firstly, keypoint detection is performed on the input video sequence to extract keypoint information of human posture. After visualization processing and alignment with the skeleton, a preliminary skeleton sequence is obtained, laying the foundation for subsequent processing. Secondly, a diffusion probability model is used to add noise to the keypoint sequence, superimposing Gaussian noise of different degrees over time. The longer the time, the more noise is added. This strategy can better simulate dynamic changes in motion, capture the temporal dependence of motion, and enable the model to have a more detailed perception of motion changes. Finally, using the noisy keypoint sequence as a condition, under the guidance of semantic features, the diffusion probability model is used to predict and remove noise to generate the final skeleton sequence. The semantic feature guidance covers high-level information such as action category and emotional state, ensuring that the generated action matches specific semantics and improving the rationality and diversity of the action. Through gradual denoising, the model recovers a clear skeleton sequence, ensuring that the generated action conforms to physical laws and is consistent with semantics. The motion synthesis apparatus provided in this application embodiment effectively combines keypoint detection, diffusion probability model, and semantic guidance technology, which can solve the problems of robustness, long sequence fluency, and action diversity in traditional motion generation, providing a more natural and controllable motion generation solution for fields such as virtual human synthesis and film and animation.
[0080] It should be noted that the specific implementation of the motion synthesis device in this application embodiment is similar to the specific implementation of the motion synthesis method in this application embodiment. Please refer to the description in the method section for details, which will not be repeated here.
[0081] Figure 6 This is a schematic diagram of the structure of the electronic device 300 according to an embodiment of this application.
[0082] like Figure 6 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage section 302 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0083] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.
[0084] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the electronic device of this application.
[0085] It should be noted that the computer-readable medium shown in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electronic device, apparatus, or device that is electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0086] In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used or combined with an electronic device, apparatus, or device by instructions. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use or combined with an electronic device, apparatus, or device by instructions. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.
[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of processing and receiving devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based electronic device that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0088] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be housed in a processor, which executes the program to implement the action synthesis method.
[0089] The video sequence is detected to obtain a key point sequence, which is then visualized and aligned with the human skeleton to obtain a preliminary skeleton sequence.
[0090] Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence over time, with longer time intervals receiving more Gaussian noise; and
[0091] Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
[0092] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs, which, when used by one or more processors, execute the action synthesis method described in this application:
[0093] The video sequence is detected to obtain a key point sequence, which is then visualized and aligned with the human skeleton to obtain a preliminary skeleton sequence.
[0094] Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence over time, with longer time intervals receiving more Gaussian noise; and
[0095] Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
[0096] In another aspect, this application also provides a computer program product, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer program product stores one or more programs, which, when used by one or more processors, execute the action synthesis method described in this application:
[0097] The video sequence is detected to obtain a key point sequence, which is then visualized and aligned with the human skeleton to obtain a preliminary skeleton sequence.
[0098] Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence over time, with longer time intervals receiving more Gaussian noise; and
[0099] Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
[0100] The above description is merely a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made based on the content of this application's specification and drawings under the concept of this application, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.
Claims
1. A method for motion synthesis, characterized in that, include: The video sequence is detected to obtain a key point sequence, which is then visualized and aligned with the human skeleton to obtain a preliminary skeleton sequence. Using a diffusion probability model, Gaussian noise of varying degrees is superimposed on the keypoint sequence over time, with longer time intervals receiving more Gaussian noise; and Using the noisy keypoint sequence as a condition, guided by semantic features, the diffusion probability model is used to predict noise, and the final skeleton sequence is obtained after denoising.
2. The motion synthesis method according to claim 1, characterized in that, After detecting keypoint sequences in the video sequence, performing visualization processing, and aligning them with a human skeleton to obtain a preliminary skeleton sequence, the method further includes: The preliminary skeleton sequence is encoded.
3. The motion synthesis method according to claim 2, characterized in that, The process of employing a diffusion probability model to superimpose different levels of Gaussian noise onto the keypoint sequence over time, with longer time intervals resulting in more Gaussian noise, further includes: The initial skeleton sequence after noise addition is mapped to the latent vector space of the diffusion probability model through positional encoding and a linear layer; and Time is encoded as the number of sampling steps and mapped to the latent vector space of the diffusion probability model through a linear layer.
4. The motion synthesis method according to claim 3, characterized in that, After encoding time as the number of sampling steps and mapping it to the latent vector space of the diffusion probability model via a linear layer, the method further includes: If the time exceeds the preset time, a distributed guidance method will be used for motion synthesis.
5. The motion synthesis method according to claim 4, characterized in that, After the initial skeleton sequence encoded with added noise is positionally encoded and mapped to the latent vector space of the diffusion probability model through a linear layer, the following steps are also included: The scale and style features of the characters in the video are learned and encoded, mapped to the latent vector space of the diffusion probability model through a linear layer, and concatenated with the features of the initial skeleton sequence, and then mapped to the latent vector space of the diffusion probability model through another linear layer.
6. The motion synthesis method according to claim 5, characterized in that, The step of using the noisy keypoint sequence as a condition, guided by semantic features, employing the diffusion probability model to predict noise, and obtaining the final skeleton sequence after denoising includes: The noisy keypoint sequence is used as a condition, and a semantic vector representing semantic features is superimposed. The diffusion probability model is used to predict the noise, and the final skeleton sequence is obtained after denoising.
7. The motion synthesis method according to claim 1, characterized in that, Before obtaining the final skeleton sequence after denoising by using the noisy keypoint sequence as a condition and predicting noise using the diffusion probability model under the guidance of semantic features, the method further includes: Separate audio from video and extract audio features; and Rhythm-prosodic features and MFCC spectral features are introduced and concatenated with the audio features, and then mapped to the latent vector space of the diffusion probability model through a linear layer.
8. A motion synthesis device, characterized in that, include: The preliminary skeleton sequence acquisition module is used to detect key point sequences in video sequences, perform visualization processing, and align them with human skeletons to obtain preliminary skeleton sequences. The noise addition module is used to apply Gaussian noise of varying degrees to the keypoint sequence over time using a diffusion probability model, wherein the longer the time interval, the more Gaussian noise is added; and The final skeleton sequence generation module is used to take the noisy keypoint sequence as a condition, and under the guidance of semantic features, use the diffusion probability model to predict noise, and obtain the final skeleton sequence after denoising.
9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the action synthesis method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for implementing the action synthesis method according to any one of claims 1-7.