Method for generating action video, training method and device of diffusion model

Abstract

The present disclosure provides a method for generating action video, a method for training diffusion model and equipment, which belongs to the technical field of computer. The method comprises: obtaining image noise data corresponding to target audio and target audio related data, the target audio related data comprising target audio or audio feature data of the target audio; inputting the target audio related data and the image noise data into the trained diffusion model, so that the diffusion model performs image denoising and restoration processing on the image noise data based on the target audio related data, to obtain predicted action data corresponding to the target audio; and generating an action video corresponding to the target audio based on the target audio and the predicted action data. By using the present disclosure, the matching degree of the predicted action data and the target audio is improved, and the integrity of the generated action video is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for generating motion videos, a method for training a diffusion model, and an apparatus. Background Technology

[0002] The technique of generating matching dance moves based on given music is widely used in scenarios where virtual digital humans dance. Currently, the method for generating dance moves involves: dividing the given music into multiple audio segments; using a trained audio feature extraction model to extract features from each audio segment; obtaining feature information for each segment; then, comparing the similarity between the feature information of each music and the feature information of each audio segment in a database storing feature information from multiple music tracks and matching dance move data for each track; determining the music with the highest similarity to the audio segment's feature information; and then directly identifying the dance move data corresponding to the music with the highest similarity as the dance move data for that audio segment; finally, concatenating the dance move data from each audio segment to obtain the dance move data for the given music.

[0003] While the above method can determine the dance movement data corresponding to a given piece of music, it actually first calculates the similarity between the audio segments of the given music and other music, then pairs the audio segments with existing dance movement data based on the similarity, and finally splices multiple dance movement data together. However, since the audio segments of the given music are usually not exactly the same as the music with the highest similarity, and the music with the highest similarity to each audio segment usually does not belong to the same piece of music, the dance movement data obtained by this pairing and splicing is not very compatible with the given music, and the transitions between dance movements are also relatively unnatural. Summary of the Invention

[0004] This disclosure provides a method for generating motion videos that can improve the matching degree between music and dance motion data.

[0005] Firstly, a method for generating motion videos is provided, the method comprising:

[0006] Obtain image noise data and target audio-related data corresponding to the target audio, wherein the target audio-related data includes the target audio or audio feature data of the target audio;

[0007] The target audio-related data and the image noise data are input into the trained diffusion model, so that the diffusion model performs image denoising and restoration processing on the image noise data based on the target audio-related data to obtain the predicted action data corresponding to the target audio.

[0008] Based on the target audio and the predicted motion data, an action video corresponding to the target audio is generated.

[0009] Secondly, a training method for a diffusion model is provided, the method comprising:

[0010] Acquire training sample data, wherein the training sample data includes sample audio-related data and baseline action data corresponding to the sample audio, and the sample audio-related data includes the sample audio or the audio feature data of the sample audio;

[0011] Noise is added to the baseline motion data corresponding to the sample audio to obtain the noise-added motion data corresponding to the sample audio.

[0012] The noisy action data and the sample audio related data are input into the diffusion model to be trained to obtain the predicted action data corresponding to the sample audio.

[0013] Based on the baseline action data and the predicted action data, the diffusion model to be trained is trained, and if the preset training termination condition is met, the trained diffusion model is obtained.

[0014] Thirdly, an apparatus for generating motion video is provided, the apparatus comprising:

[0015] The first acquisition module is used to acquire image noise data and target audio-related data corresponding to the target audio, wherein the target audio-related data includes the target audio or audio feature data of the target audio;

[0016] The first processing module is used to input the target audio-related data and the image noise data into the trained diffusion model, so that the diffusion model performs image denoising and restoration processing on the image noise data based on the target audio-related data to obtain the predicted action data corresponding to the target audio.

[0017] The generation module is used to generate an action video corresponding to the target audio based on the target audio and the predicted action data.

[0018] Fourthly, a training device for a diffusion model is provided, the device comprising:

[0019] The second acquisition module is used to acquire training sample data, wherein the training sample data includes sample audio-related data and baseline action data corresponding to the sample audio, and the sample audio-related data includes the sample audio or the audio feature data of the sample audio;

[0020] The second processing module is used to add noise to the baseline motion data corresponding to the sample audio to obtain the noise-added motion data corresponding to the sample audio.

[0021] The third processing module is used to input the noisy action data and the sample audio related data into the diffusion model to be trained, so as to obtain the predicted action data corresponding to the sample audio.

[0022] The training module is used to train the diffusion model to be trained based on the baseline action data and the predicted action data. If the preset training termination condition is met, the trained diffusion model is obtained.

[0023] Fifthly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the instruction being loaded and executed by the processor to implement the operations performed by any of the above-described methods for generating motion videos or training diffusion models.

[0024] In a sixth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the instruction being loaded and executed by a processor to implement the operations performed by the action video generation method or the diffusion model training method described above.

[0025] In a seventh aspect, a computer program product is provided, the computer program product including at least one instruction, the at least one instruction being loaded and executed by a processor to implement the operations performed by any of the above-described methods for generating motion videos or training diffusion models.

[0026] The beneficial effects of the technical solution provided in this disclosure are as follows: the image noise data and target audio-related data corresponding to the target audio are input into the trained diffusion model, so that the diffusion model can perform image denoising and restoration processing on the image noise data based on the target audio-related data, thereby obtaining the predicted action data corresponding to the target audio. Since the inverse diffusion process of the diffusion model is a Markov chain implemented by a neural network, the noisy data is gradually restored to the original data, which can achieve a strong data fitting ability. Therefore, when predicting the action data corresponding to the target audio, the image noise data can be restored to action data that fits the target audio better, so that the obtained predicted action data is closer to the distribution of real data, improving the matching degree between the predicted action data and the target audio, and the transition between actions is more natural. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating a method for generating motion video according to an embodiment of this disclosure;

[0029] Figure 2 This is a schematic flowchart of a method for determining predicted action data provided in an embodiment of this disclosure;

[0030] Figure 3 This is a schematic flowchart of a method for determining predicted action data provided in an embodiment of this disclosure;

[0031] Figure 4 This is a schematic flowchart of a training method for a diffusion model provided in an embodiment of this disclosure;

[0032] Figure 5 This is a schematic diagram of the structure of a motion video generation device provided in an embodiment of this disclosure;

[0033] Figure 6 This is a schematic diagram of the structure of a training device for a diffusion model provided in an embodiment of this disclosure;

[0034] Figure 7 This is a structural block diagram of a terminal provided in an embodiment of this disclosure;

[0035] Figure 8 This is a structural block diagram of a server provided in an embodiment of this disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0037] This disclosure provides a method for generating motion video, which can be implemented by a computer device. The computer device can be a terminal or server, and the terminal can be a desktop computer, laptop computer, tablet computer, mobile phone, etc.

[0038] Computer equipment may include processors, memory, communication components, etc.

[0039] The processor can be a central processing unit (CPU). The processor can be used to read instructions and process data, such as acquiring image noise data and target audio-related data corresponding to the target audio, determining the predicted action data corresponding to the target audio, generating action video corresponding to the target audio based on the target audio and predicted action data, and so on.

[0040] The memory can be various types of volatile or non-volatile memory, such as solid-state drives (SSDs) and dynamic random access memory (DRAM). The memory can be used for data storage, for example, storing acquired image noise data, storing acquired target audio-related data, storing data corresponding to the trained diffusion model, storing predicted action data corresponding to the determined target audio, storing generated action videos corresponding to the target audio, and so on.

[0041] Communication components can be wired network connectors, wireless fidelity (WiFi) modules, Bluetooth modules, cellular communication modules, etc. These components can be used to transmit data with other devices.

[0042] The motion video generation method provided in this disclosure can be applied to music or video applications. It determines motion data matching an existing audio clip, then uses this motion data to move a virtual character, generating a motion video that matches the audio. For example, if the audio is music, the determined motion data can be dance motion data. Based on this dance motion data, the virtual character is driven to dance, generating a dance image. Combining the music and dance image yields a dance video that matches the music and dance. As another example, if the audio is a dialogue, the determined motion data can be data on facial expression changes of the person speaking the dialogue. Based on this facial expression change data, the virtual character speaks and makes corresponding expressions. Combining the dialogue audio with the data of the virtual character speaking and making corresponding expressions yields a video of the virtual character speaking the dialogue. Of course, the motion video generation method can also be applied to other reasonable applications or other reasonable scenarios, and this disclosure does not limit this application.

[0043] Figure 1 This is a flowchart illustrating a method for generating motion video according to an embodiment of this disclosure. See also... Figure 1 This embodiment includes:

[0044] 101. Obtain the image noise data and target audio related data corresponding to the target audio.

[0045] In implementation, when it is necessary to generate a corresponding motion video based on the target audio, the target audio-related data can be determined first. In the embodiments of this disclosure, the target audio-related data includes the target audio or the audio feature data of the target audio. For example, the target audio-related data can be the target audio itself, or it can be the audio feature data obtained after feature extraction of the target audio, etc. Of course, the target audio-related data can also include other reasonable data related to the target audio, and the embodiments of this disclosure do not specifically limit this.

[0046] When the target audio-related data includes the target audio, the method for obtaining the target audio-related data is to obtain multiple target audio frames from the target audio. These multiple target audio frames can be all the audio frames included in the target audio. Alternatively, the multiple target audio frames can be a subset of audio frames obtained after downsampling all the audio frames included in the target audio. This reduces the amount of data processing. The sampling frequency of the downsampling process can be set according to requirements to obtain different numbers of target audio frames. For example, it can be set to acquire 30 audio frames per second as target audio frames, etc. This disclosure does not specifically limit the specific number of target audio frames.

[0047] When the target audio-related data includes the audio feature data of the target audio, the method for obtaining the target audio-related data is as follows: input the target audio into the trained audio feature extraction model to obtain the audio feature information corresponding to each audio frame in the target audio, and then obtain the audio feature information corresponding to multiple target audio frames in the target audio as the audio feature data of the target audio.

[0048] Similarly, multiple target audio frames can be all the audio frames included in the target audio, and the target audio related data includes the audio feature information corresponding to each audio frame included in the target audio.

[0049] Alternatively, multiple target audio frames can be a subset of audio frames in the target audio. After obtaining the audio feature information corresponding to each audio frame in the target audio, these multiple audio feature information can be downsampled to obtain the audio feature information corresponding to a subset of audio frames. The audio feature information corresponding to this subset of audio frames is then the audio feature information corresponding to the target audio frames. In this way, the accuracy of the audio feature information can be guaranteed while effectively reducing the amount of data processing.

[0050] In this embodiment of the disclosure, the audio feature extraction model can be a Jukebox model, which is pre-trained with 1 million songs and contains powerful representational features that can be used for music prediction tasks. This improves the accuracy of the audio feature information of the obtained target audio, thereby improving the matching degree between the subsequently obtained predicted action data and the target audio.

[0051] Of course, the audio feature extraction model can also be other existing feature extraction models, and this disclosure does not specifically limit them.

[0052] The image noise data in this embodiment can be image noise that follows a certain distribution function, obtained based on a preset mathematical model. This image noise data does not represent any graphics or images. In this embodiment, the image noise data can be any type of image noise obtained randomly, such as Gaussian noise. In this embodiment, using Gaussian noise for subsequent processing can transform complex data distributions into isotropic Gaussian distributions, thereby improving the training stability and accuracy of the diffusion model described later, and further improving the accuracy of the predicted action data output by the diffusion model. Of course, the image noise data in this embodiment can also be other types of image noise, such as Poisson noise, etc. This embodiment does not specifically limit this.

[0053] 102. Input the target audio-related data and image noise data into the trained diffusion model so that the diffusion model can perform image denoising and restoration processing on the image noise data based on the target audio-related data to obtain the predicted action data corresponding to the target audio.

[0054] In practice, the acquired target audio-related data and image noise data can be input into the trained diffusion model. The diffusion model will fuse the information in the target audio-related data with the image noise data, and perform image denoising and restoration processing on the image noise data, so that the image noise data carries information related to the target audio-related data, thereby obtaining predicted action data that matches the target audio-related data.

[0055] The predicted motion data corresponding to the target audio can include the motion data of the image frame corresponding to each target audio frame. That is, for each target audio frame, there is motion data of its corresponding image frame to match it.

[0056] Furthermore, the motion data of the image frame can include motion data of multiple human key points in the image frame. That is, the predicted motion data corresponding to the target audio can include motion data of multiple human key points in the image frame corresponding to each target audio frame. In this way, based on the motion data of multiple human key points in these multiple image frames, it can be used to drive the virtual character to complete the corresponding actions, thereby obtaining the image data corresponding to each image frame.

[0057] In one possible implementation, multiple human keypoints may include any combination of skeletal joints, such as the existing 24 skeletal joints, etc., which are not specifically limited in this disclosure. The motion data of each human keypoint may include at least one of the following: coordinate values ​​in the forward / backward direction, coordinate values ​​in the up / down direction, coordinate values ​​in the left / right direction, pitch angle, yaw angle, and roll angle. Of course, the motion data may also be other data that can drive the virtual character to complete the action, which are not specifically limited in this disclosure.

[0058] In one possible implementation, the model structure of the trained diffusion model can be any reasonable machine learning model, such as a 3-layer self-attention model Transformer, etc., and this disclosure does not specifically limit it.

[0059] 103. Based on the target audio and predicted motion data, generate the motion video corresponding to the target audio.

[0060] In practice, after obtaining the predicted motion data corresponding to the target audio, the virtual character can be driven to complete the corresponding action based on the motion data of multiple human key points in each image frame included in the predicted motion data, thereby obtaining the image data corresponding to each image frame.

[0061] Then, based on the correspondence between each target audio frame and image frame, the target audio and the image data corresponding to multiple image frames can be synthesized to obtain the motion data corresponding to the target audio. When playing the motion video, the image data corresponding to the image frame will be displayed at the playback time point of each target audio frame, obtaining the image data matching each target audio frame. The combination of the target audio and the image data is the motion video corresponding to the target audio.

[0062] In summary, in this embodiment, the image noise data and target audio-related data corresponding to the target audio are input into the trained diffusion model. This enables the diffusion model to perform image denoising and restoration processing on the image noise data based on the target audio-related data, thereby obtaining the predicted action data corresponding to the target audio. Since the inverse diffusion process of the diffusion model is a Markov chain implemented by a neural network, it gradually restores the noisy data to the original data, achieving a strong data fitting capability. Therefore, when predicting the action data corresponding to the target audio, the image noise data can be restored to action data that better fits the target audio, making the obtained predicted action data closer to the distribution of real data, improving the matching degree between the predicted action data and the target audio, and making the transitions between actions more natural.

[0063] The following section introduces a method for using a trained diffusion model to perform image denoising and restoration on target audio-related data and image noise data:

[0064] In one possible implementation, the trained diffusion model can include multiple diffusion sub-models. That is, the parameter combinations in the trained diffusion sub-models have multiple different values, each value corresponds to a trained diffusion sub-model, and each diffusion sub-model can perform image denoising and restoration processing on the input data, but the degree of image denoising and restoration processing performed by each diffusion sub-model is different.

[0065] Multiple diffusion sub-models can be sorted based on the degree of processing, and a corresponding number can be set for each diffusion sub-model in the sort. This number is the time step information corresponding to each diffusion sub-model. For example, if the diffusion model includes 100 diffusion sub-models, the time step information corresponding to these 100 diffusion sub-models can be 0, 1, 2, 3...97, 98, 99.

[0066] In order to select a specific diffusion sub-model from the multiple diffusion sub-models included in the diffusion model, each time step information can be encoded and converted to obtain time step encoded data for each time step information. This time step encoded data is data that can be recognized by the diffusion model. Thus, if it is necessary to use the diffusion sub-model corresponding to a certain time step information for data processing, the time step encoded data corresponding to that time step information can be input into the diffusion model, and the diffusion model can obtain the corresponding diffusion sub-model based on the time step encoded data.

[0067] The encoding conversion process can be any possible method, and this embodiment does not limit it. For example, [sin(t),cos(t)] can be calculated as the time step encoded data corresponding to the time step information, where t is the time step information.

[0068] In step 102, the following processing can be performed: determine at least one target time step information corresponding to the target audio, and use the time step encoded data of each target time step information as the target time step encoded data of the target audio; based on the correspondence between the diffusion sub-model and the time step encoded data, determine the diffusion sub-model corresponding to at least one target time step encoded data; input the target audio related data and image noise data into the diffusion sub-model corresponding to at least one target time step encoded data to obtain the predicted action data corresponding to the target audio.

[0069] In practice, when generating motion video from target audio, at least one target time step can be determined first. That is, the diffusion sub-model to be used in subsequent processing is determined by selecting the target time step from multiple time step information.

[0070] After determining the target time step information, the time step encoded data of the target time step information can be used as the target time step encoded data of the target audio. Then, based on the correspondence between the diffusion sub-model and the time step encoded data, at least one diffusion sub-model corresponding to the target time step encoded data can be determined. In this way, the diffusion sub-model to be used later can be obtained from the multiple diffusion sub-models included in the diffusion model.

[0071] Then, the target audio-related data and image noise data can be input into the diffusion sub-model corresponding to at least one target time step encoded data to obtain the output predicted action data corresponding to the target audio.

[0072] In this embodiment of the disclosure, the number of target time step information corresponding to the target audio is selected according to the requirements. When determining the target time step information corresponding to the target audio, only one of the multiple time step information can be determined as the target time step information, or multiple time step information of a portion of the multiple time step information can be determined as the target time step information, or all of the multiple time step information can be determined as the target time step information. The selection can be made according to the requirements, and this embodiment of the disclosure does not limit it.

[0073] When selecting target time step information, factors such as the complexity of the target audio, the time required for data processing, and the accuracy required for the motion video can be considered.

[0074] For example, when the target audio has a strong rhythm and changes rapidly, a relatively large number of time steps can be selected as the target time step information. This means that a relatively large number of diffusion sub-models will be used to process the target audio-related data and image noise data to obtain more accurate predicted action data. When the target audio has a gentle rhythm and changes slowly, a relatively small number of time steps can be selected as the target time step information. This means that a relatively small number of diffusion sub-models will be used to process the target audio-related data and image noise data. In this way, while ensuring that the obtained predicted action data is relatively accurate, the amount of data processing is effectively reduced, thereby reducing data processing time and improving efficiency.

[0075] When there is only one target time step information corresponding to the target audio, that is, there is only one target time step encoded data and one diffusion sub-model corresponding to the target time step encoded data.

[0076] During step 102, the target audio-related data and image noise data can be directly input into the diffusion sub-model corresponding to the target time step encoded data to obtain the predicted action data corresponding to the target audio output by the diffusion sub-model corresponding to the target time step encoded data. The image noise data can be randomly generated image noise obtained based on a preset mathematical model.

[0077] When there are multiple target time step information and target time step encoded data corresponding to the target audio, the number of image noise data corresponding to the target audio can also be set to multiple, and the number of image noise data corresponding to the target audio is equal to the number of target time step information and the number of target time step encoded data.

[0078] Each image noise data can be associated with a diffusion sub-model corresponding to different target time step encoded data (also known as a diffusion sub-model corresponding to different target time step information). That is, each diffusion sub-model corresponding to the target time step encoded data corresponds to an image noise data. The image noise data corresponding to the diffusion sub-model corresponding to each target time step encoded data can all be image noise randomly generated based on a preset mathematical model.

[0079] During step 102, the target time-step encoded data, target audio-related data, and image noise data of the diffusion sub-models corresponding to multiple target time-step encoded data can all be input into the diffusion model. The diffusion model will determine the corresponding diffusion sub-model based on the input target time-step encoded data and use the diffusion sub-model corresponding to the target time-step encoded data to perform data processing, thereby finally outputting the predicted action data corresponding to the target audio. See [link to relevant documentation]. Figure 2 The data processing method based on multiple diffusion sub-models can be as follows:

[0080] 201. Sort the diffusion sub-models corresponding to multiple target time step encoded data in descending order of the target time step information.

[0081] In practice, since different diffusion sub-models perform image denoising and restoration to different degrees, and when training multiple diffusion sub-models in the diffusion model, they are sorted according to the degree of processing and time step information is used to sort these multiple diffusion sub-models. Therefore, at this time, the diffusion sub-models corresponding to the selected multiple target time step encoded data can be sorted in descending order of their corresponding time step information in order to proceed with the subsequent processing steps 202 and 203.

[0082] The relationship between the processing level of the diffusion sub-model and the numerical value of its corresponding time step information can be set according to requirements. The correspondence between the diffusion sub-model and the time step information can be: as the value of the time step information increases, the processing level of the diffusion sub-model corresponding to the time step information also increases, or as the value of the time step information increases, the processing level of the diffusion sub-model corresponding to the time step information decreases. This embodiment does not impose specific limitations on this.

[0083] 202. For the first diffusion sub-model in the sorting, the target audio-related data and the image noise data corresponding to the first diffusion sub-model are used as the input data of the first diffusion sub-model to obtain the action data output by the first diffusion sub-model.

[0084] In practice, since each target time step encoded data corresponds to a different image noise data, the image noise data corresponding to the first diffusion sub-model in the sorting determined in step 201 can be obtained.

[0085] Then, the target audio-related data and the image noise data corresponding to the first diffusion sub-model are input into the first diffusion sub-model to obtain the motion data output by the diffusion sub-model. The motion data may include the motion data of multiple human key points in the image frame corresponding to each target audio frame in the target audio.

[0086] 203. For non-first diffusion sub-models in the ranking (excluding the first diffusion sub-model), fuse the action data output by the previous diffusion sub-model and the image noise data corresponding to the non-first diffusion sub-model to obtain fused action data. Use the fused action data and target audio-related data as input data for the non-first diffusion sub-model to obtain the action data output by the non-first diffusion sub-model. Use the action data output by the last non-first diffusion sub-model in the ranking as the predicted action data corresponding to the target audio.

[0087] Here, "non-first diffusion sub-model" refers to any diffusion sub-model other than the first diffusion sub-model in the sorting process among the diffusion sub-models corresponding to the target time step encoded data. In implementation, after obtaining the motion data output by the first diffusion sub-model in the sorting process through step 202, this motion data can be fused with the image noise data corresponding to the second diffusion sub-model in the sorting process to obtain the fused motion data corresponding to the second diffusion sub-model.

[0088] The motion data output by the first diffusion sub-model is obtained by predicting the target audio-related data and can match the target audio to a certain extent. After fusing this motion data with the image noise data corresponding to the second diffusion sub-model, the motion data can be blurred to a certain extent to obtain fused motion data.

[0089] Then, the fused motion data and target audio-related data can be input into the second diffusion sub-model in the sorting process to obtain the motion data output by the second diffusion sub-model. When the second diffusion sub-model processes the fused motion data and target audio-related data, it can perform image denoising and restoration processing on the fused motion data based on the target audio-related data, thereby outputting the motion data corresponding to the second diffusion sub-model. In this way, the second diffusion sub-model performs image denoising and restoration processing on the fused motion data carrying relevant information from the motion data output by the first diffusion sub-model, enabling deeper processing of the target audio-related data. This allows the motion data output by the second diffusion sub-model to reconstruct motion data that better matches the target audio based on the fused motion data, thus improving the accuracy of the obtained motion data.

[0090] Then, the motion data output by the second diffusion sub-model in the sorting process is fused with the image noise data corresponding to the third diffusion sub-model to obtain the fused motion data corresponding to the third diffusion sub-model. Then, the fused motion data corresponding to the third diffusion sub-model and the target audio-related data can be input into the third diffusion sub-model to obtain the motion data output by the third diffusion sub-model. This motion data is an image denoising and restoration process performed on the fused motion data corresponding to the third diffusion sub-model. It can restore motion data that matches the target audio better based on the fused motion data corresponding to the third diffusion sub-model, thereby further improving the accuracy of the obtained motion data.

[0091] By analogy, multiple diffusion sub-models corresponding to the target time step encoded data are used to process the target audio-related data and the target time step encoded data, and the action data corresponding to the target audio is restored step by step on the image noise data until the action data output by the last non-first diffusion sub-model in the sorting is obtained, which is the final predicted action data output by the diffusion model.

[0092] In this way, by using the diffusion sub-model corresponding to the data encoded by multiple target time steps for prediction, the image noise data can be restored into motion data that is more in line with the target audio, thereby improving the matching degree between the predicted motion data and the target audio, and making the transitions between actions more natural.

[0093] In one possible implementation, the fusion processing performed in step 203 above can be achieved by directly adding the image noise data corresponding to the non-first diffusion sub-model and the action data output by the previous diffusion sub-model element-wise to obtain the fused action data.

[0094] In another possible implementation, the fusion processing performed in step 203 above can be performed by adding noise to the motion data output by the previous diffusion sub-model based on the target time step information corresponding to the non-first diffusion sub-model and the image noise data corresponding to the non-first diffusion sub-model, to obtain fused motion data.

[0095] In practice, based on the different values ​​of the target time step information corresponding to the non-first diffusion sub-model, different degrees of noise processing can be applied to the image noise data corresponding to the non-first diffusion sub-model and the motion data output by the previous diffusion sub-model.

[0096] When predicting motion data, the diffusion sub-models are used to process the data in descending order of the target time step information. As the diffusion sub-models corresponding to multiple target time steps are processed step by step, the motion data output by the diffusion sub-model corresponding to the smaller the value of the target time step information is, the more accurate the motion data is and the higher the matching degree with the target audio. Therefore, as the value of the target time step information corresponding to the non-first diffusion sub-model becomes smaller and smaller, the noise addition of the image noise data corresponding to the non-first diffusion sub-model and the motion data output by the previous diffusion sub-model can become smaller and smaller. This results in less and less relevant data of image noise data carried in the fused motion data and more and more relevant data of motion data output by the previous diffusion sub-model, so as to obtain motion data that matches the target audio better, thereby improving the matching degree between the final predicted motion data and the target audio.

[0097] In one possible implementation, the noise addition process based on the target time step information corresponding to the non-first diffusion sub-model, the image noise data corresponding to the non-first diffusion sub-model, and the motion data output by the previous diffusion sub-model can be as follows:

[0098] Based on the target time step information corresponding to the non-first diffusion sub-model, a first weight value corresponding to the action data output by the previous diffusion sub-model and a second weight value corresponding to the image noise data corresponding to the non-first diffusion sub-model are determined. The first weight value is negatively correlated with the target time step information corresponding to the non-first diffusion sub-model, and the second weight value is positively correlated with the target time step information corresponding to the non-first diffusion sub-model. The first weight value is multiplied by the action data output by the previous diffusion sub-model to obtain the weighted action data. The second weight value is multiplied by the image noise data corresponding to the non-first diffusion sub-model to obtain the weighted image noise data. The weighted action data and the weighted image noise data are added together to obtain the fused action data.

[0099] In practice, the target time step information corresponding to the non-first diffusion sub-model can be input into the preset algorithm to obtain the first weight value corresponding to the action data output by the previous diffusion sub-model and the second weight value corresponding to the image noise data corresponding to the non-first diffusion sub-model.

[0100] The first weight value is negatively correlated with the target time step information, while the second weight value is positively correlated with the target time step information. That is, as the target time step information decreases, the first weight value increases and the second weight value decreases. In this way, as the target time step information decreases, the fused action data carries more and more relevant data from the action data output by the previous diffusion sub-model and less and less relevant data from the image noise data, so as to obtain action data that matches the target audio better, thereby improving the matching degree between the final predicted action data and the target audio.

[0101] In one possible implementation, the first weight value and the second weight value can be pre-set first weight value and second weight value corresponding to different target time step information, or the first weight value and the second weight value can be calculated based on the following formula (1):

[0102]

[0103] Where t1 represents the target time step information corresponding to the non-first diffusion sub-model, and the target time step information corresponding to the non-first diffusion sub-model is a positive integer; β i β is the preset noise intensity coefficient for the target time step information i corresponding to the non-first diffusion sub-model. i It is positively correlated with i; Q1 is the first weight value; Q2 is the second weight value.

[0104] Of course, the method for calculating the first weight value and the second weight value can be any other reasonable method, and this disclosure does not specifically limit it.

[0105] This disclosure also provides a method for training a diffusion model, such as... Figure 4 As shown, the method includes:

[0106] 401. Obtain training sample data.

[0107] The training sample data includes sample audio-related data and baseline motion data corresponding to the sample audio. The sample audio-related data includes sample audio or audio feature data of the sample audio.

[0108] In one possible implementation, the method for obtaining training sample data may be: obtaining multiple types of action videos from an action video database; for each type of action video, segmenting the action video to obtain multiple action video segments corresponding to the action video; for each action video segment, determining the baseline action data corresponding to the sample audio based on the multiple image frames included in the action video segment, and determining the sample audio related data based on the multiple audio frames included in the action video segment.

[0109] The selection method for the motion video database can be as follows: the motion video database can be any existing video database that meets the requirements. This database can include various types of motion videos. For example, the motion video database can be the publicly available dataset AIST++. AIST++ is a large-scale multimodal 3D dance motion dataset, constructed from the AIST dance video database. This dataset provides 3D human keypoint annotations and camera parameters for 10.1 million images, covering 30 different themes in 9 views. This dataset contains 5.2 hours of 3D dance motion, divided into 1408 sequences, covering 10 dance types. Each type includes multi-view videos taken from known camera positions and directions. Using the AIST++ dataset to obtain training sample data improves the comprehensiveness and accuracy of the training sample data, thereby improving the comprehensiveness and accuracy of the subsequently determined diffusion model after training.

[0110] Action videos are typically quite long. Therefore, they can be segmented based on a preset video length to obtain multiple action video segments. Each action video segment includes corresponding audio and action data, where the audio can be used as sample audio. The preset video length can be any reasonable length, such as 30 seconds, 40 seconds, 60 seconds, etc., and this embodiment does not limit it.

[0111] In one possible implementation, the method for determining the reference motion data corresponding to the sample audio can be: determining multiple sample image frames among multiple image frames included in the action video segment; for each sample image frame, acquiring motion data of multiple human key points in the sample image frame as the reference motion data corresponding to the sample audio.

[0112] In practice, the sample image frames can be all the image frames included in the action video clip, or they can be a subset of image frames obtained by downsampling all the image frames. This disclosure does not limit the scope of the sample image frames.

[0113] In one possible implementation, multiple human key points may include any combination of skeletal joints, such as the existing 24 skeletal joints, etc., which are not specifically limited in this disclosure.

[0114] The motion data of each human body key point may include at least one of the following: coordinate values ​​in the front-back direction, coordinate values ​​in the up-down direction, coordinate values ​​in the left-right direction, pitch angle, yaw angle, and roll angle. Of course, the motion data may also be other data that can drive the virtual character to complete the action. This disclosure embodiment does not specifically limit this.

[0115] In one possible implementation, the method for determining the relevant data of the sample audio is any one or more of the following:

[0116] The first method: When the sample audio-related data includes sample audio, multiple sample audio frames from the action video clip can be obtained as sample audio-related data. These sample audio frames can be all the audio frames included in the action video clip, or they can be the audio frames in the action video clip that correspond to the sample image frames.

[0117] The second approach: When the sample audio-related data includes the audio feature data of the sample audio, the following processing can be performed: input multiple audio frames included in the action video segment into the audio feature extraction model to obtain the audio feature information corresponding to the multiple audio frames, and determine the audio feature information corresponding to the multiple audio frames as the sample audio-related data. In this embodiment, the audio feature extraction model can be the Jukebox model, which is pre-trained using 1 million songs and contains powerful representational features that can be used for music prediction tasks, improving the accuracy of the obtained target audio's audio feature information, thereby improving the matching degree between the subsequently obtained predicted action data and the target audio. Of course, the audio feature extraction model can also be other existing feature extraction models, and this embodiment does not specifically limit it.

[0118] The third method involves identifying the audio frame corresponding to each sample image frame within the multiple audio frames of an action video clip as a sample audio frame. The audio feature information of each sample audio frame is then obtained as sample audio-related data. In implementation, multiple audio frames from the action video clip are input into an audio feature extraction model, which outputs the audio feature information corresponding to each audio frame. Within the multiple audio frames of the action video clip, the audio frame corresponding to the sample image frame is identified as the sample audio frame. Then, the audio feature information corresponding to these multiple sample audio frames is determined as sample audio-related data.

[0119] 402. Noise is added to the baseline motion data corresponding to the sample audio to obtain the noise-added motion data corresponding to the sample audio.

[0120] In implementation, the baseline motion data is noise-added to obscure its actual meaning, which is then used to train the diffusion model. The noise-adding method can involve first acquiring image noise data that does not represent any graphics or images, and then calculating a weighted sum of the image noise data and the baseline motion data with different weight values ​​to obtain the noisy motion data.

[0121] The image noise data can be any type of image noise obtained randomly, such as Gaussian noise. In this embodiment, using Gaussian noise for subsequent processing can transform the complex data distribution into an isotropic Gaussian distribution, thereby achieving stable training of the diffusion model. Correspondingly, the diffusion model can achieve stable training by optimizing the variational lower bound of the data likelihood (e.g., variational lower bound, or ELBO (Evidence Lower Bound)), thus having a significant advantage in fitting the true data distribution. Of course, the image noise data in this embodiment can also be other types of image noise, such as Poisson noise, etc., and this embodiment does not specifically limit it.

[0122] 403. Input the noisy action data and the relevant data of the sample audio into the diffusion model to be trained to obtain the predicted action data corresponding to the sample audio.

[0123] In one possible implementation, the model structure of the diffusion model to be trained can be any reasonable machine learning model, such as a 3-layer self-attention model Transformer, etc., and this disclosure does not specifically limit it.

[0124] 404. Based on the baseline action data and the predicted action data, train the diffusion model to be trained. If the preset training termination condition is met, the trained diffusion model is obtained.

[0125] In implementation, the baseline action data and predicted action data can be input into the loss function to calculate the loss value. Then, based on the loss value, the diffusion model to be trained is trained. When a preset training termination condition is met, the resulting diffusion model is determined as the trained diffusion model. For example, the method for calculating the loss value can be to calculate the minimum mean square error between the baseline action data and the predicted action data to obtain the loss value. Then, gradient descent is performed on the diffusion model to be trained based on this loss value. The optimizer used in the model is Adam, and the learning rate can be 0.01. When the loss value converges, the training ends, and the trained diffusion sub-model is obtained. The preset training termination condition can be: the determined loss value is less than a preset loss value threshold, or the number of training iterations reaches a preset number of iterations threshold, etc., or other reasonable termination conditions. This embodiment of the disclosure does not limit these conditions.

[0126] Using the training method of the diffusion model provided in this disclosure, the predicted motion data output by the trained motion generation data can have a high degree of matching with the audio, thereby obtaining a more fitting motion video.

[0127] In one possible implementation, the diffusion model to be trained may include multiple diffusion sub-models to be trained, and the parameter combinations in each diffusion sub-model to be trained may have different values.

[0128] Similarly, these multiple diffusion sub-models to be trained can be sorted, and corresponding time step information can be set for them according to the sorting. That is, each diffusion sub-model to be trained corresponds to different sample time step information. For example, when the diffusion model to be trained includes 50 diffusion sub-models, the sample time step information corresponding to these 50 diffusion sub-models to be trained can be 0, 1, 2, ... 48, 49 respectively.

[0129] In order to select a specific diffusion sub-model from the diffusion model, the information of each sample time step can be encoded and transformed to obtain the sample time step encoded data of each sample time step. This sample time step encoded data is data that can be recognized by the diffusion model to be trained. Thus, if it is necessary to use the diffusion sub-model to be trained corresponding to a certain sample time step for data processing, the sample time step encoded data corresponding to that sample time step can be input into the diffusion model to be trained. The diffusion model to be trained can obtain the corresponding diffusion sub-model to be trained based on the sample time step encoded data.

[0130] The encoding conversion process can be any possible method, and this embodiment does not limit it. For example, [sin(t),cos(t)] can be calculated as the sample time step encoded data corresponding to the sample time step information, where t is the sample time step information.

[0131] In one possible implementation, when the diffusion model to be trained includes multiple diffusion sub-models to be trained, the training process can be performed as follows: a preset number of training sample data are obtained from the training set, where the preset number is the number of time step information. These preset number of training sample data are randomly matched with multiple sample time step encoded data to obtain training sample data corresponding to each sample time step encoded data. Then, the diffusion sub-model to be trained corresponding to each sample time step encoded data can be trained based on the training sample data corresponding to each sample time step encoded data, thereby obtaining the trained diffusion sub-model corresponding to each sample time step encoded data.

[0132] In other words, for each sample time step encoded data corresponding to the diffusion sub-model to be trained, the training sample data for the diffusion sub-model to be trained includes sample audio-related data and baseline action data corresponding to the sample audio. When training the diffusion sub-model corresponding to the sample time step encoded data, noise can be added to the baseline action data corresponding to the sample audio to obtain noisy action data corresponding to the sample audio; the noisy action data and sample audio-related data are input into the diffusion sub-model to be trained to obtain the predicted action data corresponding to the sample audio output by the diffusion sub-model to be trained; based on the baseline action data and predicted action data corresponding to the sample audio, the diffusion sub-model to be trained is trained, and if the preset training termination condition is met, the trained diffusion sub-model is obtained.

[0133] For example, if the diffusion model to be trained includes 100 diffusion sub-models to be trained, then the number of sample time step encoded data is 100. When obtaining training sample data, 100 training sample data can be randomly obtained from the training set first, and then these 100 training sample data can be randomly matched with 100 time step information, so that each training sample data corresponds to a time step information.

[0134] Then, these 100 training sample data can be used to train the diffusion sub-models corresponding to their respective time step encoding data. After these 100 training sample data have trained their respective diffusion sub-models, another 100 training sample data can be randomly obtained from the training set, in addition to the above 100 training sample data. These 100 training sample data are then randomly matched with the 100 time step information to obtain the training sample data corresponding to each time step encoding data. These 100 training sample data are then used to train the diffusion sub-models corresponding to their respective time step encoding data. This process is repeated, ensuring that the training times of multiple diffusion sub-models are relatively even when training multiple diffusion sub-models.

[0135] In one possible implementation, during step 402, the following processing can be performed: obtain the image noise data corresponding to each diffusion sub-model to be trained; for each diffusion sub-model to be trained, based on the sample time step information corresponding to the diffusion sub-model to be trained and the image noise data corresponding to the diffusion sub-model to be trained, perform noise processing on the baseline motion data corresponding to the sample audio of the diffusion sub-model to be trained, to obtain the noise-added motion data corresponding to the diffusion sub-model to be trained.

[0136] The image noise data can be image noise that follows a certain distribution function, obtained based on a preset mathematical model. This image noise data does not represent any graphics or images. In this embodiment, the image noise data can be any type of image noise obtained randomly, such as Gaussian noise, Poisson noise, etc. This embodiment does not specifically limit it.

[0137] In practice, based on the different numerical values ​​of the sample time step information corresponding to each diffusion sub-model to be trained, the baseline action data corresponding to the diffusion sub-model to be trained can be subjected to different degrees of noise addition processing, so that the trained diffusion sub-model can perform different degrees of image denoising and restoration processing on the input data.

[0138] In one possible implementation, for each diffusion sub-model to be trained, the method of applying different degrees of noise based on the numerical values ​​of the sample time step information corresponding to the diffusion sub-model to be trained can be as follows: Based on the sample time step information corresponding to the diffusion sub-model to be trained, determine the third weight value corresponding to the baseline action data and the fourth weight value corresponding to the image noise data of the diffusion sub-model to be trained, wherein the third weight value is negatively correlated with the sample time step information and the fourth weight value is positively correlated with the sample time step information; multiply the third weight value with the baseline action data to obtain the weighted baseline action data; multiply the fourth weight value with the image noise data to obtain the weighted image noise data; add the weighted baseline action data and the weighted image noise data to obtain the noisy action data corresponding to the diffusion sub-model to be trained.

[0139] Thus, since the matching degree between the reference motion data and the sample audio is high, based on the above noise addition method, the larger the sample time step information, the smaller the third weight value corresponding to the reference motion data and the larger the fourth weight value corresponding to the image noise data. In this way, the obtained noise-added motion data is more blurred.

[0140] In one possible implementation, the third and fourth weight values ​​can be pre-set weight values ​​corresponding to different sample time step information, or they can be calculated based on the following formula (2):

[0141]

[0142] Where t2 represents the sample time step information, and the sample time step information is a positive integer; β i β is the preset noise intensity coefficient for time step i. i It is positively correlated with i; Q3 is the third weight value; Q4 is the fourth weight value.

[0143] Of course, the method for calculating the third and fourth weight values ​​can be any other reasonable method, and this disclosure does not specifically limit it.

[0144] In this way, by using noisy action data with different levels of noise to train the diffusion sub-models corresponding to the time step encoded data of different samples, the trained diffusion sub-models corresponding to the time step encoded data of different samples can be obtained.

[0145] After obtaining multiple trained diffusion sub-models using the above training method, they can be applied to the above action video generation method. Multiple trained diffusion sub-models corresponding to the target time step encoded data can be selected to process the target audio related data and image noise data in sequence, thereby obtaining action data with a higher matching degree with the target audio step by step, and finally obtaining predicted action data with a high matching degree with the target audio.

[0146] This disclosure provides an apparatus for generating motion video, such as... Figure 5 As shown, the device includes:

[0147] The first acquisition module 510 is used to acquire image noise data and target audio related data corresponding to the target audio, wherein the target audio related data includes the target audio or audio feature data of the target audio;

[0148] The first processing module 520 is used to input the target audio-related data and the image noise data into the trained diffusion model, so that the diffusion model performs image denoising and restoration processing on the image noise data based on the target audio-related data to obtain the predicted action data corresponding to the target audio.

[0149] The generation module 530 is used to generate an action video corresponding to the target audio based on the target audio and the predicted action data.

[0150] In one possible implementation, the trained diffusion model includes multiple diffusion sub-models, each of which corresponds to encoded data at different time steps.

[0151] The first processing module 520 is used for:

[0152] Determine at least one target time step information corresponding to the target audio, and determine the diffusion sub-model corresponding to the at least one target time step information based on the correspondence between the diffusion sub-model and the time step information;

[0153] The target audio-related data and the image noise data are input into the diffusion sub-model corresponding to the at least one target time step information to obtain the predicted action data corresponding to the target audio.

[0154] In one possible implementation, the number of target time step information corresponding to the target audio and the number of image noise data corresponding to the target audio are equal and both are multiple, and each image noise data corresponds to a diffusion sub-model corresponding to a different target time step information.

[0155] The first processing module 520 is used for:

[0156] Sort the diffusion sub-models corresponding to the target time step information in descending order of the target time step information;

[0157] For the first diffusion sub-model in the sorting, the target audio-related data and the image noise data corresponding to the first diffusion sub-model are used as the input data of the first diffusion sub-model to obtain the action data output by the first diffusion sub-model;

[0158] For any non-first diffusion sub-model in the sorting (excluding the first diffusion sub-model), the action data output by the preceding diffusion sub-model and the corresponding image noise data are fused to obtain fused action data. The fused action data and the target audio-related data are then used as input data for the non-first diffusion sub-model to obtain the action data output by the non-first diffusion sub-model. Finally, the action data output by the last non-first diffusion sub-model in the sorting is used as the predicted action data corresponding to the target audio.

[0159] In one possible implementation, the first processing module 520 is used for:

[0160] Based on the target time step information corresponding to the non-first diffusion sub-model and the image noise data corresponding to the non-first diffusion sub-model, the motion data output by the previous diffusion sub-model is noise-added to obtain the fused motion data.

[0161] In one possible implementation, the first processing module 520 is configured to:

[0162] Based on the target time step information corresponding to the non-first diffusion sub-model, a first weight value corresponding to the action data output by the previous diffusion sub-model and a second weight value corresponding to the image noise data corresponding to the non-first diffusion sub-model are determined. The first weight value is negatively correlated with the target time step information corresponding to the non-first diffusion sub-model, and the second weight value is positively correlated with the target time step information corresponding to the non-first diffusion sub-model.

[0163] Multiply the first weight value by the action data output by the previous diffusion sub-model to obtain the weighted action data; multiply the second weight value by the image noise data corresponding to the non-first diffusion sub-model to obtain the weighted image noise data;

[0164] The weighted motion data and the weighted image noise data are added together to obtain the fused motion data.

[0165] In one possible implementation, the first processing module 520 is configured to:

[0166] The first weight value and the second weight value are calculated based on the following formulas;

[0167]

[0168] Wherein, t1 is the target time step information corresponding to the non-first diffusion sub-model, and the target time step information corresponding to the non-first diffusion sub-model is a positive integer;

[0169] βi is a preset noise intensity coefficient when the target time step information is i, which is not the first diffusion sub-model, and βi is positively correlated with i.

[0170] Q1 is the first weight value;

[0171] Q2 is the second weight value.

[0172] In one possible implementation, the predicted motion data corresponding to the target audio includes motion data of multiple human key points in the image frame corresponding to each target audio frame in the target audio.

[0173] It should be noted that the motion video generation device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the motion video generation device and the motion video generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0174] This disclosure provides a training apparatus for a diffusion model, such as... Figure 6 As shown, the device includes:

[0175] The second acquisition module 610 is used to acquire training sample data, wherein the training sample data includes sample audio related data and baseline action data corresponding to the sample audio, and the sample audio related data includes the sample audio or the audio feature data of the sample audio;

[0176] The second processing module 620 is used to add noise to the baseline motion data corresponding to the sample audio to obtain the noise-added motion data corresponding to the sample audio.

[0177] The third processing module 630 is used to input the noisy action data and the sample audio related data into the diffusion model to be trained, so as to obtain the predicted action data corresponding to the sample audio.

[0178] The training module 640 is used to train the diffusion model to be trained based on the baseline action data and the predicted action data. If the preset training termination condition is met, the trained diffusion model is obtained.

[0179] In one possible implementation, the second acquisition module 610 is used for:

[0180] Multiple types of action videos are acquired, and for each type of action video, the action video is segmented to obtain multiple action video segments corresponding to the action video;

[0181] For each of the action video segments, based on the multiple image frames included in the action video segment, the baseline motion data corresponding to the sample audio is determined, and based on the multiple audio frames included in the action video segment, the sample audio related data is determined.

[0182] In one possible implementation, the second acquisition module 610 is used for:

[0183] Multiple sample image frames are determined from the multiple image frames included in the action video segment, and for each sample image frame, motion data of multiple human key points in the sample image frame are obtained as the reference motion data corresponding to the sample audio.

[0184] In the multiple audio frames included in the action video segment, the audio frame corresponding to each sample image frame is determined as a sample audio frame; the audio feature information of each sample audio frame in the multiple sample audio frames is obtained as the sample audio related data.

[0185] It should be noted that the diffusion model training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the diffusion model. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the diffusion model training device and the diffusion model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0186] Figure 7 A structural block diagram of a terminal 700 provided in an exemplary embodiment of this disclosure is shown. The terminal can be a computer device as described in the above embodiments. The terminal 700 can be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 700 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0187] Typically, terminal 700 includes a processor 701 and a memory 702.

[0188] Processor 701 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 701 may be implemented using at least one hardware form selected from DSP (digital signal processing), FPGA (field-programmable gate array), and PLA (programmable logic array). Processor 701 may also include a main processor and a coprocessor. The main processor, also known as a CPU (central processing unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 701 may integrate a GPU (graphics processing unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 701 may also include an AI (artificial intelligence) processor, which is used to handle computational operations related to machine learning.

[0189] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 702 are used to store at least one instruction, which is executed by the processor 701 to implement the motion video generation method or the diffusion model training method provided in the method embodiments of this disclosure.

[0190] In some embodiments, the terminal 700 may also optionally include a peripheral device interface 703 and at least one peripheral device. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 704, a display screen 705, a camera 706, an audio circuit 707, a positioning component 708, and a power supply 709.

[0191] Peripheral device interface 703 can be used to connect at least one I / O (input / output) related peripheral device to processor 701 and memory 702. In some embodiments, processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 701, memory 702 and peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0192] The radio frequency (RF) circuit 704 is used to receive and transmit RF (radio frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 704 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (wireless fidelity) networks. In some embodiments, the RF circuit 704 may also include circuitry related to NFC (near field communication), which is not limited in this disclosure.

[0193] Display screen 705 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 705 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 701 for processing. In this case, display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 705, which serves as the front panel of terminal 700; in other embodiments, there may be at least two display screens 705, respectively disposed on different surfaces of terminal 700 or in a folded design; in still other embodiments, display screen 705 may be a flexible display screen, disposed on a curved or folded surface of terminal 700. Furthermore, display screen 705 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 705 may be made of materials such as LCD (liquid crystal display) or OLED (organic light-emitting diode).

[0194] The camera assembly 706 is used to acquire images or videos. Optionally, the camera assembly 706 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (virtual reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0195] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 701 for processing, or input to the radio frequency circuit 704 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 700. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0196] The positioning component 708 is used to determine the current geographic location of the positioning terminal 700 in order to enable navigation or LBS (location-based service). The positioning component 708 can be a positioning component based on GPS (global positioning system), BeiDou system, Grenadin system, or Galileo system.

[0197] The power supply 709 is used to power the various components in the terminal 700. The power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0198] In some embodiments, the terminal 700 further includes one or more sensors 710. The one or more sensors 710 include, but are not limited to: an accelerometer 711, a gyroscope 712, a pressure sensor 713, a fingerprint sensor 714, an optical sensor 715, and a proximity sensor 716.

[0199] Accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal 700. For example, accelerometer 711 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 701 can control display screen 705 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 711. Accelerometer 711 can also be used for games or for acquiring user motion data.

[0200] The gyroscope sensor 712 can detect the orientation and rotation angle of the terminal 700. The gyroscope sensor 712, in conjunction with the accelerometer sensor 711, can collect 3D motion data from the user on the terminal 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0201] The pressure sensor 713 can be disposed on the side bezel of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 713 is disposed on the side bezel of the terminal 700, it can detect the user's grip signal on the terminal 700, and the processor 701 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is disposed on the lower layer of the display screen 705, the processor 701 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 705. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0202] The fingerprint sensor 714 is used to collect a user's fingerprint. The processor 701 identifies the user based on the fingerprint collected by the fingerprint sensor 714, or vice versa. When the user's identity is identified as trusted, the processor 701 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 714 can be located on the front, back, or side of the terminal 700. When the terminal 700 has a physical button or manufacturer logo, the fingerprint sensor 714 can be integrated with the physical button or manufacturer logo.

[0203] An optical sensor 715 is used to collect ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 based on the ambient light intensity collected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity collected by the optical sensor 715.

[0204] The proximity sensor 716, also known as a distance sensor, is typically located on the front panel of the terminal 700. The proximity sensor 716 is used to detect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from a screen-on state to a screen-off state; when the proximity sensor 716 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from a screen-off state to a screen-on state.

[0205] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on terminal 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0206] Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this disclosure. The server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 801 and one or more memories 802. The memory 802 stores at least one instruction, which is loaded and executed by the processor 801 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0207] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including instructions that can be executed by a processor in a terminal to complete the motion video generation method or the diffusion model training method in the above embodiments. The computer-readable storage medium may be non-transitory. For example, the computer-readable storage medium may be ROM (read-only memory), RAM (random access memory), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0208] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0209] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices) involved in this disclosure are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the "target audio" and "training sample data" involved in this disclosure were obtained with full authorization.

[0210] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A method for generating motion video, characterized in that, The method includes: Acquire multiple image noise data and target audio-related data corresponding to the target audio, wherein the target audio-related data includes the target audio or audio feature data of the target audio; Determine multiple target time step information corresponding to the target audio, wherein the number of multiple target time step information and the number of multiple image noise data are equal; Based on the correspondence between the diffusion sub-models and time step information in the trained diffusion model, the diffusion sub-model corresponding to the target time step information is determined, wherein each diffusion sub-model corresponds to a different time step information, and each image noise data corresponds to a different diffusion sub-model corresponding to the target time step information. Sort the diffusion sub-models corresponding to the target time step information in descending order of the target time step information; For the first diffusion sub-model in the sorting, the target audio-related data and the image noise data corresponding to the first diffusion sub-model are used as the input data of the first diffusion sub-model to obtain the action data output by the first diffusion sub-model; For each of the non-first diffusion sub-models in the ranking (excluding the first diffusion sub-model), based on the target time step information corresponding to the non-first diffusion sub-model, a first weight value corresponding to the action data output by the preceding diffusion sub-model and a second weight value corresponding to the image noise data of the non-first diffusion sub-model are determined. The first weight value is negatively correlated with the target time step information of the non-first diffusion sub-model, and the second weight value is positively correlated with the target time step information of the non-first diffusion sub-model. The first weight value is multiplied by the action data output by the preceding diffusion sub-model to obtain weighted action data. The second weight value is multiplied by the image noise data corresponding to the non-first diffusion sub-model to obtain weighted image noise data. The weighted action data and the weighted image noise data are added together to obtain fused action data. The fused action data and the target audio-related data are used as input data for the non-first diffusion sub-model to obtain the action data output by the non-first diffusion sub-model. The action data output by the last non-first diffusion sub-model in the ranking is determined as the predicted action data corresponding to the target audio. Based on the target audio and the predicted motion data, an action video corresponding to the target audio is generated.

2. The method according to claim 1, characterized in that, The step of determining the first weight value corresponding to the action data output by the previous diffusion sub-model and the second weight value corresponding to the image noise data of the non-first diffusion sub-model based on the target time step information corresponding to the non-first diffusion sub-model includes: The first weight value and the second weight value are calculated based on the following formulas; Among them, the This refers to the target time step information corresponding to a non-first diffusion sub-model, where the target time step information corresponding to the non-first diffusion sub-model is a positive integer; The target time step information for the non-first diffusion sub-model is as follows: The preset noise intensity coefficient at that time, the With the There is a positive correlation; the aforementioned The first weight value; This is the second weight value.

3. The method according to claim 1, characterized in that, The predicted motion data corresponding to the target audio includes motion data of multiple human key points in the image frame corresponding to each target audio frame in the target audio.

4. A training method for a diffusion model, characterized in that, The method includes: The diffusion model to be trained includes multiple diffusion sub-models to be trained, which are sorted, and the corresponding sample time step information is set for the diffusion sub-models to be trained according to the sorting. Acquire the training sample data corresponding to each sample time step information and the image noise data corresponding to each diffusion sub-model to be trained. The training sample data includes sample audio related data and the baseline action data corresponding to the sample audio. The sample audio related data includes the sample audio or the audio feature data of the sample audio. For each diffusion sub-model to be trained, based on the sample time step information corresponding to the diffusion sub-model to be trained, a third weight value corresponding to the baseline action data and a fourth weight value corresponding to the image noise data of the diffusion sub-model to be trained are determined. The third weight value is negatively correlated with the sample time step information, and the fourth weight value is positively correlated with the sample time step information. The third weight value is multiplied by the baseline action data to obtain the weighted baseline action data. The fourth weight value is multiplied by the image noise data to obtain the weighted... The image noise data is then processed; the weighted baseline action data and the weighted image noise data are added together to obtain the noisy action data corresponding to the diffusion sub-model to be trained; the noisy action data and the sample audio related data corresponding to the diffusion sub-model to be trained are input into the diffusion sub-model to be trained to obtain the predicted action data corresponding to the sample audio output by the diffusion sub-model to be trained; based on the baseline action data corresponding to the sample audio and the predicted action data, the diffusion sub-model to be trained is trained, and if the preset training termination condition is met, the trained diffusion sub-model is obtained.

5. The method according to claim 4, characterized in that, The step of obtaining the training sample data corresponding to each sample time step information includes: Multiple types of action videos are acquired, and for each type of action video, the action video is segmented to obtain multiple action video segments corresponding to the action video; For each of the action video segments, based on the multiple image frames included in the action video segment, the baseline motion data corresponding to the sample audio is determined, and based on the multiple audio frames included in the action video segment, the sample audio related data is determined.

6. The method according to claim 5, characterized in that, The process of determining baseline motion data corresponding to the sample audio based on multiple image frames included in the motion video clip, and determining sample audio-related data based on multiple audio frames included in the motion video clip, includes: Multiple sample image frames are determined from the multiple image frames included in the action video segment, and for each sample image frame, motion data of multiple human key points in the sample image frame are obtained as the reference motion data corresponding to the sample audio. In the multiple audio frames included in the action video segment, the audio frame corresponding to each sample image frame is determined as a sample audio frame; the audio feature information of each sample audio frame in the multiple sample audio frames is obtained as the sample audio related data.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to perform the operations performed by the motion video generation method as described in any one of claims 1 to 3 or the diffusion model training method as described in any one of claims 4 to 6.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to perform the operation of the motion video generation method as described in any one of claims 1 to 3 or the diffusion model training method as described in any one of claims 4 to 6.

9. A computer program product, characterized in that, The computer program product includes at least one instruction, which is loaded and executed by a processor to perform the operation of the motion video generation method as described in any one of claims 1 to 3 or the diffusion model training method as described in any one of claims 4 to 6.

Images