Motion synthesis method, device, equipment, system and program product

Abstract

The embodiment of the invention provides an action synthesis method. The method comprises the following steps: determining a corresponding key point sequence of an input video; then noise adding processing is executed based on the key point sequence; in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to execute a segmented training process, the first stage is carried out based on audio features, and the second stage is carried out based on semantic features in a hidden space of the first stage; the trained diffusion probability model can be used for synthesizing object actions. By applying the diffusion probability model obtained by training, a diversified action synthesis process can be realized, and since a stable sequence structure is continuously generated in a hidden space in a first stage in a second stage in a training process, semantic features are described in a refined manner on the basis of audio features, and the accuracy of action synthesis is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an action synthesis method, device, equipment, system and program product. Background Art

[0002] In the field of human-computer interaction, motion information is an important tool for conveying information in human communication, and is also a key technology for developing realistic avatars in movies, games, virtual social environments, and interactions with social robots.

[0003] Generally, a skeleton sequence can be used as a condition to generate a person that matches the skeleton sequence. For example, virtual human synthesis in multiple scenarios such as dance video generation, speeches, press conferences, and live streaming of goods.

[0004] However, the current motion synthesis process is based on training by specific people, and the generated motions have limitations, which can easily lead to motion convergence and affect the accuracy of motion synthesis. Summary of the invention

[0005] In view of this, the present application provides an action synthesis method, which can improve the accuracy of action synthesis.

[0006] According to a first aspect of an embodiment of the present application, a method for action synthesis is provided, which can be applied to a system or program including an action synthesis function in a terminal device, and specifically includes:

[0007] inputting the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information;

[0008] During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

[0009] Optionally, in some possible implementations, the training process of the diffusion probability model includes:

[0010] Acquire an input video, and perform key point detection on each video frame in the input video to obtain a key point sequence;

[0011] Performing noise configuration on the key point sequence in time order to obtain a noise sequence;

[0012] Extracting feature information corresponding to the input video, the feature information including audio features corresponding to each of the video frames and semantic features associated with the audio features;

[0013] In the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform a training process, and the training process includes a first stage and a second stage. The first stage is performed based on the audio features, and the second stage is performed in the latent space of the first stage based on the semantic features.

[0014] Optionally, in some possible implementations, in the process of denoising the noise sequence, using the noise sequence and the feature information as conditions to guide the diffusion probability model to perform a training process includes:

[0015] In the process of denoising the noise sequence, the noise sequence and the feature information are input into the diffusion probability model, the diffusion probability model includes at least one first diffusion module and at least one second diffusion module, and the first diffusion module and the second diffusion module have a corresponding relationship;

[0016] Executing the first stage of training process based on the first diffusion module to obtain first loss information;

[0017] Inputting the output features of each of the first diffusion modules into the corresponding second diffusion module, so that the second diffusion module uses the output features as conditions to perform the second stage training process based on the semantic features to obtain second loss information;

[0018] The diffusion probability model is trained according to the first loss information and the second loss information.

[0019] Optionally, in some possible implementations, the performing the first stage training process based on the first diffusion module to obtain first loss information includes:

[0020] splicing the audio feature, the time feature and the object identification feature in the feature information to obtain a spliced ​​feature;

[0021] Transforming the concatenated features into a latent space through a linear layer to obtain latent space features;

[0022] The latent space features are input into the first diffusion module, so that the first diffusion module performs the first stage training process based on the self-attention mechanism to obtain the first loss information.

[0023] Optionally, in some possible implementations, inputting the noise sequence and the feature information into the diffusion probability model includes:

[0024] Acquire batch data corresponding to a denoising process, the batch data comprising a guide frame and a training frame, the guide frame being determined based on a plurality of adjacent preceding training frames;

[0025] The noise sequence and the feature information corresponding to the batch data are input into the diffusion probability model.

[0026] Optionally, in some possible implementations, obtaining batch data corresponding to a denoising process includes:

[0027] If the batch data is the first batch, determining the number of frames corresponding to the guide frame;

[0028] Superimposing the number of frames corresponding to the guide frame and the number of frames corresponding to the training frame to obtain a target number;

[0029] Frame extraction is performed during denoising based on the target quantity to obtain the batch data.

[0030] Optionally, in some possible implementations, extracting feature information corresponding to the input video includes:

[0031] Determine a video frame corresponding to the input video to extract audio data corresponding to the video frame;

[0032] Extracting features from the audio data based on preset dimensions to obtain audio features, wherein the preset dimensions include rhythm, prosody and spectrum;

[0033] Determining a semantic feature associated with the audio feature;

[0034] Obtaining a time sequence corresponding to the video frame to obtain a time feature;

[0035] Determining a target object indicated in the video frame, and extracting an object identification feature corresponding to the target object;

[0036] The audio feature, the semantic feature, the time feature and the object identification feature are combined to obtain the feature information.

[0037] Optionally, in some possible implementations, determining the semantic feature associated with the audio feature includes:

[0038] Get the preset action set;

[0039] Marking the start and end time in the video frame based on the actions in the preset action set;

[0040] Configure the position code according to the start and end times;

[0041] A semantic feature associated with the audio feature is obtained based on the position code and the corresponding action.

[0042] Optionally, in some possible implementations, determining the target object indicated in the video frame and extracting an object identification feature corresponding to the target object includes:

[0043] determining a target object indicated in the video frame;

[0044] Detecting the display ratio of the target object in the video frame to obtain scale information;

[0045] Obtaining style information corresponding to the target object;

[0046] The scale information and the style information are combined to obtain an object identification feature corresponding to the target object.

[0047] Optionally, in some possible implementations, the acquiring the style information corresponding to the target object includes:

[0048] Determining a learnable vector corresponding to the target object, wherein a weight parameter of the learnable vector is adjusted based on a training process of a diffusion probability model;

[0049] Corresponding style information is configured based on the learnable vector.

[0050] Optionally, in some possible implementations, the object action is used to configure a virtual agent, and a method for inputting the action information includes tag setting, text input, or voice input.

[0051] According to a second aspect of an embodiment of the present application, there is provided an action synthesis device, comprising:

[0052] a synthesis unit, configured to input the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information;

[0053] During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

[0054] Optionally, in some possible implementations, the training process of the diffusion probability model includes: obtaining an input video, and performing key point detection on each video frame in the input video to obtain a key point sequence; configuring the key point sequence with noise in chronological order to obtain a noise sequence; extracting feature information corresponding to the input video, the feature information including audio features corresponding to each of the video frames, and semantic features associated with the audio features; in the process of denoising the noise sequence, using the noise sequence and the feature information as conditions to guide the diffusion probability model to execute a training process, the training process including a first stage and a second stage, the first stage being performed based on the audio features, and the second stage being performed based on the semantic features in the latent space of the first stage.

[0055] Optionally, in some possible implementations, in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to execute a training process, including: in the process of denoising the noise sequence, the noise sequence and the feature information are input into the diffusion probability model, the diffusion probability model includes at least one first diffusion module and at least one second diffusion module, and the first diffusion module and the second diffusion module have a corresponding relationship; based on the first diffusion module, the first stage of the training process is executed to obtain first loss information; the output features of each of the first diffusion modules are input into the corresponding second diffusion module, so that the second diffusion module uses the output features as conditions to execute the second stage of the training process based on the semantic features to obtain second loss information; the diffusion probability model is trained according to the first loss information and the second loss information.

[0056] Optionally, in some possible implementations, the first stage training process is executed based on the first diffusion module to obtain the first loss information, including: splicing the audio features, time features and object identification features in the feature information to obtain the spliced ​​features; performing feature transformation of the spliced ​​features to the latent space through a linear layer to obtain the latent space features; inputting the latent space features into the first diffusion module, so that the first diffusion module executes the first stage training process based on the self-attention mechanism to obtain the first loss information.

[0057] Optionally, in some possible implementations, inputting the noise sequence and the feature information into the diffusion probability model includes: obtaining batch data corresponding to a denoising process, the batch data including a guide frame and a training frame, the guide frame being determined based on a plurality of adjacent preceding training frames; and inputting the noise sequence and the feature information corresponding to the batch data into the diffusion probability model.

[0058] Optionally, in some possible implementations, batch data corresponding to a denoising process is obtained, including: if the batch data is the first batch, determining the number of frames corresponding to the guide frame; superimposing the number of frames corresponding to the guide frame and the number of frames corresponding to the training frame to obtain a target number; and performing frame extraction during the denoising process based on the target number to obtain the batch data.

[0059] Optionally, in some possible implementations, extracting feature information corresponding to the input video includes: determining a video frame corresponding to the input video to extract audio data corresponding to the video frame; performing feature extraction on the audio data based on preset dimensions to obtain audio features, the preset dimensions including rhythm, rhyme and spectrum; determining semantic features associated with the audio features; acquiring a time series corresponding to the video frame to obtain a time feature; determining a target object indicated in the video frame and extracting an object identification feature corresponding to the target object; and combining the audio features, semantic features, time features and the object identification features to obtain the feature information.

[0060] Optionally, in some possible implementations, determining the semantic features associated with the audio features includes: obtaining a preset action set; marking the start and end times in the video frame based on the actions in the preset action set; configuring position coding according to the start and end times; and obtaining the semantic features associated with the audio features based on the position coding and the corresponding actions.

[0061] Optionally, in some possible implementations, determining the target object indicated in the video frame and extracting object identification features corresponding to the target object include: determining the target object indicated in the video frame; detecting the display ratio of the target object in the video frame to obtain scale information; acquiring style information corresponding to the target object; and combining the scale information and the style information to obtain object identification features corresponding to the target object.

[0062] Optionally, in some possible implementations, obtaining the style information corresponding to the target object includes: determining a learnable vector corresponding to the target object, wherein a weight parameter of the learnable vector is adjusted based on a training process of a diffusion probability model; and configuring corresponding style information based on the learnable vector.

[0063] Optionally, in some possible implementations, the object action is used to configure a virtual agent, and a method for inputting the action information includes tag setting, text input, or voice input.

[0064] According to a third aspect of an embodiment of the present application, there is provided an action synthesis device, including an input and output component and a processor;

[0065] The input and output components are used to obtain request information;

[0066] The processor is used to synthesize the action corresponding to the request information obtained by the input-output component by executing the action synthesis method as described in the first aspect or any one of the first aspects.

[0067] According to a fourth aspect of an embodiment of the present application, there is provided an action synthesis system, including an interactive client and a server;

[0068] The interactive client is used to obtain request information and send the request information to the server, and to display the action synthesis result output by the server;

[0069] The server is used to synthesize the action corresponding to the request information obtained by the interactive client by executing the action synthesis method described in the first aspect or any one of the first aspects.

[0070] According to a fifth aspect of an embodiment of the present application, a computer program product is provided, comprising: a computer program, which, when executed by a processor, implements the action synthesis method described in the first aspect or any one of the first aspects.

[0071] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0072] When training the diffusion probability model, the training input video is obtained, and the key point sequence is obtained by detecting the key points of each video frame in the input video; then the key point sequence is noise configured in time order to obtain a noise sequence; and the feature information corresponding to the input video is extracted, and the feature information includes the audio features corresponding to each video frame, and the semantic features associated with the audio features; and in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform the training process, and the training process includes the first stage and the second stage, wherein the first stage is based on the audio features, and the second stage is based on the semantic features in the latent space of the first stage. The synthesis of the object action using the above-mentioned trained diffusion probability model can realize a diverse action synthesis process, because the second stage in the training process continues to generate a stable sequence structure in the latent space of the first stage, and the semantic features are described in detail on the basis of the audio features, thereby improving the accuracy of the action synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0074] Figure 1 A diagram of the network architecture that runs the action synthesis system;

[0075] Figure 2 A flow chart of an action synthesis provided in an embodiment of the present application;

[0076] Figure 3 A flowchart of an action synthesis method provided in an embodiment of the present application;

[0077] Figure 4 A schematic diagram of a training architecture of a diffusion probability model provided in an embodiment of the present application;

[0078] Figure 5 A schematic diagram of the structure of another diffusion probability model provided in an embodiment of the present application;

[0079] Figure 6 A schematic diagram of training sampling of another action synthesis method provided in an embodiment of the present application;

[0080] Figure 7 A schematic diagram of the structure of an action synthesis device provided in an embodiment of the present application;

[0081] Figure 8 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] It should be understood that the action synthesis method provided in the present application can be applied to a system or program including an action synthesis function in a terminal device, such as a human-computer interaction application. Specifically, the action synthesis system can be run on a system such as Figure 1 In the network architecture shown in Figure 1As shown in FIG. 1 , it is a network architecture diagram of the operation of the action synthesis system. As can be seen from the figure, the action synthesis system can provide an action synthesis process with multiple information sources, that is, the action information is determined by the acquisition operation on the terminal side, and then sent to the server to generate the corresponding object action; it can be understood that Figure 1 A variety of terminal devices are shown in FIG. 1 . The terminal devices may be computer devices. In actual scenarios, more or fewer types of terminal devices may participate in the process of action synthesis. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may be involved, especially in multidisciplinary synthesis scenarios, and the specific number of servers depends on the actual scenario.

[0084] In this embodiment, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the terminal and the server can be connected to form a blockchain network, which is not limited in this application.

[0085] It can be understood that the above-mentioned action synthesis system can run on a personal mobile terminal, for example: as an application such as a human-computer interaction application, it can also run on a server, and it can also run on a third-party device to provide action synthesis to obtain the action synthesis processing result of the information source; the specific action synthesis system can be run in the above-mentioned device in the form of a program, and it can also be run as a system component in the above-mentioned device, and it can also be used as a cloud service program. The specific operation mode depends on the actual scenario and is not limited here.

[0086] In the field of human-computer interaction, motion information is an important tool for conveying information in human communication, and is also a key technology for developing realistic avatars in movies, games, virtual social environments, and interactions with social robots.

[0087] Generally, a skeleton sequence can be used as a condition to generate a person that matches the skeleton sequence. For example, virtual human synthesis in multiple scenarios such as dance video generation, speeches, press conferences, and live streaming of goods.

[0088] However, the current motion synthesis process is based on training by specific people, and the generated motions have limitations, which can easily lead to motion convergence and affect the accuracy of motion synthesis.

[0089] In order to solve the above problems, this application proposes an action synthesis method, which is applied to Figure 2 In the action synthesis process framework shown in Figure 2 As shown, it is a process architecture diagram of action synthesis provided in an embodiment of the present application, in which action information is obtained through a terminal, thereby triggering a diffusion probability model configured in a server to generate corresponding object actions based on the action information, wherein the diffusion probability model is guided by training based on the audio features and semantic features of a training video, and the training process adopts staged training, superimposing semantic features while maintaining audio action features.

[0090] It can be understood that the action synthesis method provided in the present application can be written as a program to serve as a processing logic in a hardware system, or as an action synthesis device to implement the above processing logic in an integrated or external manner. As an implementation method, the action synthesis device obtains a training input video and performs key point detection on each video frame in the input video to obtain a key point sequence; then the key point sequence is noise configured in chronological order to obtain a noise sequence; and the feature information corresponding to the input video is extracted, and the feature information includes the audio features corresponding to each video frame, and the semantic features associated with the audio features; and in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform a training process, and the training process includes a first stage and a second stage, wherein the first stage is based on audio features, and the second stage is based on semantic features in the latent space of the first stage; the trained diffusion probability model can be used to synthesize object actions. The diffusion probability model obtained by the above training can realize a diverse action synthesis process, because the second stage in the training process continues to generate a stable sequence structure in the latent space of the first stage, and the semantic features are described in detail on the basis of the audio features, thereby improving the accuracy of action synthesis.

[0091] Combined with the above process architecture, the following will introduce the action synthesis method in this application. Please refer to Figure 3 , Figure 3 This is a flowchart of an action synthesis method provided in an embodiment of the present application. The embodiment of the present application at least includes the following steps:

[0092] 301. Input the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information.

[0093] In this embodiment, the action information is the configuration information used to instruct the virtual agent to perform an action. The virtual agent of the configuration object action is generated based on the diffusion probability model; wherein, the input method of the action information includes a combination of one or more of tag setting, text input, voice input or other media content input, and after the action information is input, the skeleton sequence is used as a condition to generate a person matching the skeleton sequence. For example, virtual people are synthesized in multiple scenarios such as dance video generation, speeches, press conferences, and live broadcasts; for example, voice input generates cheerful dance movements, and corresponding background music is configured, then a virtual person performing cheerful dance movements is generated based on the diffusion probability model provided in this embodiment, and the movements of the virtual person change with the progress of the background music; it can be seen that the action information in this embodiment can be a combination of multiple media forms, and the specific combination method depends on the actual scene; in addition, the above virtual people are examples, and their specific object forms depend on the actual scene.

[0094] Specifically, the diffusion probability model of this embodiment adopts a two-stage serial action synthesis process, that is, the action synthesis process may include a first stage and a second stage. First, a first action is generated in the first stage in response to the action information, and then the semantic dimension of the first action is adjusted in the second stage, so that the second stage continues to generate a stable skeleton structure in the latent space of the first stage, thereby improving the accuracy of action generation.

[0095] In some possible scenarios, the input action information may not be configured with corresponding semantic information. In this case, the corresponding object action is directly generated according to the audio information configured in the action information.

[0096] It can be understood that compared to inputting action information and semantic information separately, this embodiment introduces semantic information for action synthesis based on the action information, and aligns the feature dimensions of the action information and the semantic information, thereby avoiding the feature loss caused by the alignment operation due to the different representation dimensions of the action information and the semantic information.

[0097] The following describes the training process of the diffusion probability model used in this embodiment.

[0098] During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the feature information corresponding to the noise sequence and the input video is used as conditions to guide the diffusion probability model to execute the training process. The feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage. The first stage is based on the audio features, and the second stage is based on the semantic features in the latent space of the first stage.

[0099] In this embodiment, the model training architecture for the training process of the diffusion probability model is as follows: Figure 4 As shown, Figure 4 A schematic diagram of a training architecture of a diffusion probability model provided in an embodiment of the present application; this embodiment adopts a diffusion probability model as the basic architecture, and the diffusion probability model training is divided into two processes: forward noise addition and reverse noise removal. Below, the training process of the diffusion probability model based on this architecture includes the following steps:

[0100] S1: Obtain an input video, and perform key point detection on each video frame in the input video to obtain a key point sequence.

[0101] In this embodiment, the input video is the training data of the diffusion probability model, in which a virtual object that performs actions based on audio data is configured; therefore, key point detection is performed on each video frame in the input video, that is, a process of performing key point detection on the virtual object that performs actions therein.

[0102] Specifically, for the encoding execution of the key point sequence detection process, taking the output result represented by 18 human key points as an example, the skeleton sequence X0 of n frames is represented by the dimension X0∈[n,18×2], where 18×2 represents the two coordinates of each point (x, y); after adding noise, it is represented as X t ∈[n,18×2], the sequence is mapped to a 256-dimensional latent space through position encoding and a linear layer.

[0103] S2: Noise the key point sequence in time order to obtain a noise sequence.

[0104] In this embodiment, the key point sequence is configured with noise in time order, that is, the forward noise adding process. Specifically, the video sequence can be first input to detect the key point sequence frame by frame using key point detection models such as dwpose, and the key point sequence can be aligned with the human skeleton after visualization; in the forward noise adding process, Gaussian noise of different degrees is superimposed on the key point sequence according to time t, where the larger t is, the more noise is added. When t=1000, the skeleton sequence becomes pure Gaussian noise. Among them, the value of t is only an example, and the specific value range depends on the actual scene.

[0105] S3: Extract feature information corresponding to the input video, where the feature information includes audio features corresponding to each video frame and semantic features associated with the audio features.

[0106] In this embodiment, the process of extracting feature information corresponding to the input video includes the process of extracting audio features and the process of extracting semantic features. Considering that the video frame has the characteristics of a time series, the feature information also includes time features. In order to further improve the richness of object actions, the features of different types of objects can be represented, so the feature information also includes object identification features, i.e., ID features.

[0107] Specifically, for the audio feature extraction process, first determine the video frame corresponding to the input video to extract the audio data corresponding to the video frame; then extract the audio data based on the preset dimension to obtain the audio feature. The preset dimension includes rhythm and spectrum. That is, n frames of input video correspond to n frames of audio. The audio condition is extracted through the WavLM model. The feature dimension is represented as X m ∈[n,1024], at the same time, in order to improve the sensitivity to audio, rhythm-related features and spectral features such as MFCC are introduced, all of the above features are cancatted in the spatial dimension, and then mapped to the 256-dimensional latent space through a linear layer.

[0108] After determining the audio features, the semantic features associated with the audio features can be determined; then the time series corresponding to the video frame is obtained to obtain the time feature, that is, the time t is an integer between 0 and 1000, representing the number of sampling steps, and the time is mapped to the 256-dimensional feature X through the linear layer ti ∈

[256] , and then repeat it n times on the frame to get X ti ∈[n,256]; further determine the target object indicated in the video frame, and extract the object identification features corresponding to the target object; and then combine the audio features, semantic features, time features and object identification features to obtain feature information.

[0109] Specifically, since the diversity and randomness of the actions outputted only by audio features are relatively large, in order to increase controllability and the synthesis of characteristic actions, a semantic vector representation is proposed. Therefore, for the determination of semantic features, a preset action set is first obtained, that is, 10 common actions are annotated in the training data set, as shown in Table 1; then the start and end times are annotated in the video frame based on the actions in the preset action set; and the position coding is configured according to the start and end times; and then the semantic features associated with the audio features are obtained based on the position coding and the corresponding actions. Among them, the start time and end time of the corresponding action in the video frame are annotated during the action annotation process, and the action feature is represented as X d ∈[n,8], using onehot encoding; according to the start time and end time of the action, the concept of time position encoding is introduced X t ∈[n,256]: For example, if there are 100 frames of skeleton sequence, and the 20th to 80th frames are greeting actions, then X t [20:80] Encoding using sine and cosine. Final semantic feature X d After position encoding and linear layer mapping to 256-dimensional latent space, it is added to the 256-dimensional position encoding.

[0110] Table 1 Intent-Action Design Table

[0111] intention action Onehot ID emphasize Raise your hand to emphasize 1 make a fist 2 Finger pinch 3 Index finger raised 4 reject Shake your head 5 Hand shaking 6 Praise Like 7 greet Wave 8

[0112] It is understandable that most of the current action generation frameworks are based on specific people, so there is no scale change of ID, so onehot is generally used to represent different styles; however, because there are various characters such as old and young in the data set, ID information is needed to characterize the character style information while also capturing the character scale information. That is, considering that the training process based on specific people is prone to poor robustness of generated actions, it cannot effectively capture the scale and style of different people; and the richness and diversity of the actions are small, and the results of each reasoning are the same. At the same time, the actions and audio rhythms cannot be strictly checked, resulting in poor rationality of the actions. Therefore, this embodiment proposes an ID feature representation that includes style and skeleton scale, that is, object identification features, for the scale and style issues of non-specific people.

[0113] Specifically, the representation of the object identification feature includes scale information indicating the range of the object and style information indicating the style of the object. That is, first determine the target object indicated in the video frame; then detect the display ratio of the target object in the video frame to obtain scale information; and obtain style information corresponding to the target object; and then combine the scale information and style information to obtain the object identification feature corresponding to the target object.

[0114] Furthermore, for the representation of style information, since the style of the object in the video may change, it is necessary to set up a dynamic adjustment process. Therefore, a learnable vector corresponding to the target object can be determined, and the weight parameters of the learnable vector are adjusted based on the training process of the diffusion probability model; then the corresponding style information is configured based on the learnable vector.

[0115] In a possible scenario, the ID feature can be designed as follows: id1 ∈

[120] represents a random frame in the current skeleton sequence, such as the display ratio of the object in the frame (half body, full body, etc.); the second part: X id2 ∈

[256] represents a learnable embedding whose weight is updated as the network is trained, allowing the model to learn its style information. Specifically, for the representation of ID features in this configuration scenario, a frame of ID features X id1 ∈

[120] linear layer maps to a 256-dimensional latent space and then id2 ∈

[256] feature concat, get X id3 ∈

[512] , and maps its linear layer to the 256-dimensional latent space to obtain the final ID feature X id ∈

[256] , so the 256-dimensional feature encodes both character scale and style information.

[0116] S4: In the process of denoising the noise sequence, the noise sequence and feature information are used as conditions to guide the diffusion probability model to perform a training process. The training process includes a first stage and a second stage. The first stage is based on audio features, and the second stage is based on semantic features in the latent space of the first stage.

[0117] In this embodiment, it is indicated that in the process of reverse denoising, the noisy key point sequence is used as a condition, and under the guidance of audio / semantic features, the noise is predicted by the difusion model, and the predicted key point sequence is obtained after denoising; in the process of this reasoning, random Gaussian noise is used as input, and the noise prediction starts at t=1000, and the final skeleton sequence is obtained through 1000 iterations of denoising.

[0118] Specifically, the training process of the diffusion probability model in the denoising process is explained in combination with the structure of the diffusion probability model, such as Figure 5 As shown, Figure 5 A schematic diagram of the structure of another diffusion probability model provided for an embodiment of the present application; the structure of the diffusion probability model of this embodiment is based on a diffusion module (DIT block) and is divided into a first stage and a second stage, wherein the design of the DIT block allows time step signals and text control signals to be concatenated with the input sequence as additional tokens, so that the original ViT (Vision Transformer) architecture can be kept unchanged, and the increased computational effort can be ignored, thereby combining the feature extraction capability of the Transformer into the diffusion probability model.

[0119] Therefore, in the process of denoising the noise sequence, the noise sequence and feature information are input into the diffusion probability model, and the diffusion probability model includes at least one first diffusion module and at least one second diffusion module. The first diffusion module and the second diffusion module have a corresponding relationship, that is, Figure 5 The number setting in is an example, and the specific DIT block number configuration depends on the actual scenario. Then, the first stage training process is performed based on the first diffusion module to obtain the first loss information; and the output features of each first diffusion module are input into the corresponding second diffusion module, so that the second diffusion module uses the output features as conditions to perform the second stage training process based on the semantic features to obtain the second loss information; and then the diffusion probability model is trained according to the first loss information and the second loss information.

[0120] Specifically, the first stage of training is guided by audio conditions, which include audio features, time features, and object identification features. That is, the audio features, time features, and object identification features in the feature information are concatenated to obtain concatenated features; then the concatenated features are transformed to latent space through a linear layer to obtain latent space features; and the latent space features are input into the first diffusion module, so that the first diffusion module performs the first stage of training based on the self-attention mechanism to obtain the first loss information.

[0121] For the first stage of training, the Figure 5 In the scenario, the audio features are concatenated in the feature dimension and time embedding (time feature) and id embedding (object identification feature); then the features are transformed into latent space through the linear layer. This process is expressed as MLP (concat (X t ,X m ,X ti ,X id )∈[n,1024])∈[n,256]: Then the latent space feature is used as a condition to feed into 4 DITblocks. DITblock is constructed with the classic self-attention mechanism. In order to ensure that the ID information remains unchanged, the ID embedding is used as a condition to embed it in the DIT block in the form of a cross attention mechanism. The last DITblock outputs the feature X o1 The head output is composed of several layers of MLP and the loss function is calculated.

[0122] Furthermore, the training process of the second stage is combined with Figure 5 In the scenario, the second stage continues to introduce semantic conditions based on the latent in the first stage, and is also encoded into 256-dimensional features through the linear layer. The process is expressed as MLP(concat(X o1 ,X d ,X ti ,X id )∈[n,1024])∈[n,256]: After 4 DIT blocks, in order not to lose the diversity of the audio in the first stage, the output features of each block in the first stage are embedded into the block of the second stage in a cross attention manner as conditions, and the two stages are jointly trained.

[0123] Furthermore, considering the problem of unsmooth generation of long sequences, this embodiment divides the long sequence into several small sequences when generating it, and then interpolates at the clip to achieve the continuity and smoothness of the action, that is, to ensure the continuity of the long-term synthetic action for the problem of long-term synthesis continuity. Specifically, the division into several small sequences can be represented as batch data, which is obtained by sampling the video frames in sequence. That is, firstly, the batch data corresponding to the denoising process is obtained, and the batch data includes a guide frame and a training frame, and the guide frame is determined based on multiple adjacent previous training frames; then the noise sequence and feature information corresponding to the batch data are input into the diffusion probability model.

[0124] In addition, considering that there is no guide frame for the first time, supplementary sampling of the guide frame is required, that is, if the batch data is the first batch, the number of frames corresponding to the guide frame is determined; then the number of frames corresponding to the guide frame and the number of frames corresponding to the training frame are superimposed to obtain the target number; and then based on the target number, frame extraction is performed during the denoising process to obtain batch data.

[0125] In one possible scenario, the batch data uses 220 frames, such as Figure 6 As shown, Figure 6 A training sampling diagram of another action synthesis method provided in an embodiment of the present application; that is, for the synthesis of long-term action sequences, taking 220 frames for each reasoning as an example, the first time, because there is no guide frame, 220 frames of random Gaussian noise are input to generate 220 frames of results; the second time, the 20 frames after the first time are used as a guide, and the 200 frames of random noise are concat in the time dimension, and then the results of 220 frames are output, and only the results of the next 200 frames are taken, and so on. Because of the above-mentioned reasoning characteristics, there are two ways to train: the first way is to generate an experiment of generating 200 frames of results with 20 frames of GT guidance; the second way is to denoise the 220 frames of random noise.

[0126] Through the above-mentioned semantic fusion process, the concepts of semantic information and temporal position encoding are introduced at the same time to ensure that the model can make corresponding actions according to semantic cues and can still make corresponding actions according to audio when there is no semantic action.

[0127] In combination with the above embodiments, it can be known that by obtaining the input video for training, and performing key point detection on each video frame in the input video to obtain a key point sequence; then the key point sequence is noise configured in chronological order to obtain a noise sequence; and the feature information corresponding to the input video is extracted, the feature information includes the audio features corresponding to each video frame, and the semantic features associated with the audio features; and in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform a training process, and the training process includes a first stage and a second stage, wherein the first stage is based on audio features, and the second stage is based on semantic features in the latent space of the first stage. The trained diffusion probability model can be used to synthesize object actions. The diffusion probability model obtained by the above training can realize a diverse action synthesis process, because the second stage in the training process continues to generate a stable sequence structure in the latent space of the first stage, and the semantic features are described in detail on the basis of the audio features, thereby improving the accuracy of action synthesis.

[0128] In order to better implement the above solution of the embodiment of the present application, the following also provides related devices for implementing the above solution. Figure 7 , Figure 7 This is a schematic diagram of the structure of an action synthesis device provided in an embodiment of the present application. The synthesis device 700 includes:

[0129] A synthesis unit 701, configured to input the action information into a diffusion probability model, so that the diffusion probability model synthesizes the object action based on the action information;

[0130] During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

[0131] Optionally, in some possible implementations, the training process of the diffusion probability model includes: obtaining an input video, and performing key point detection on each video frame in the input video to obtain a key point sequence; configuring the key point sequence with noise in chronological order to obtain a noise sequence; extracting feature information corresponding to the input video, the feature information including audio features corresponding to each of the video frames, and semantic features associated with the audio features; in the process of denoising the noise sequence, using the noise sequence and the feature information as conditions to guide the diffusion probability model to execute a training process, the training process including a first stage and a second stage, the first stage being performed based on the audio features, and the second stage being performed based on the semantic features in the latent space of the first stage.

[0132] Optionally, in some possible implementations, in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to execute a training process, including: in the process of denoising the noise sequence, the noise sequence and the feature information are input into the diffusion probability model, the diffusion probability model includes at least one first diffusion module and at least one second diffusion module, and the first diffusion module and the second diffusion module have a corresponding relationship; based on the first diffusion module, the first stage of the training process is executed to obtain first loss information; the output features of each of the first diffusion modules are input into the corresponding second diffusion module, so that the second diffusion module uses the output features as conditions to execute the second stage of the training process based on the semantic features to obtain second loss information; the diffusion probability model is trained according to the first loss information and the second loss information.

[0133] Optionally, in some possible implementations, the first stage training process is executed based on the first diffusion module to obtain the first loss information, including: splicing the audio features, time features and object identification features in the feature information to obtain the spliced ​​features; performing feature transformation of the spliced ​​features to the latent space through a linear layer to obtain the latent space features; inputting the latent space features into the first diffusion module, so that the first diffusion module executes the first stage training process based on the self-attention mechanism to obtain the first loss information.

[0134] Optionally, in some possible implementations, inputting the noise sequence and the feature information into the diffusion probability model includes: obtaining batch data corresponding to a denoising process, the batch data including a guide frame and a training frame, the guide frame being determined based on a plurality of adjacent preceding training frames; and inputting the noise sequence and the feature information corresponding to the batch data into the diffusion probability model.

[0135] Optionally, in some possible implementations, batch data corresponding to a denoising process is obtained, including: if the batch data is the first batch, determining the number of frames corresponding to the guide frame; superimposing the number of frames corresponding to the guide frame and the number of frames corresponding to the training frame to obtain a target number; and performing frame extraction during the denoising process based on the target number to obtain the batch data.

[0136] Optionally, in some possible implementations, extracting feature information corresponding to the input video includes: determining a video frame corresponding to the input video to extract audio data corresponding to the video frame; performing feature extraction on the audio data based on preset dimensions to obtain audio features, the preset dimensions including rhythm, rhyme and spectrum; determining semantic features associated with the audio features; acquiring a time series corresponding to the video frame to obtain a time feature; determining a target object indicated in the video frame and extracting an object identification feature corresponding to the target object; and combining the audio features, semantic features, time features and the object identification features to obtain the feature information.

[0137] Optionally, in some possible implementations, determining the semantic features associated with the audio features includes: obtaining a preset action set; marking the start and end times in the video frame based on the actions in the preset action set; configuring position coding according to the start and end times; and obtaining the semantic features associated with the audio features based on the position coding and the corresponding actions.

[0138] Optionally, in some possible implementations, determining the target object indicated in the video frame and extracting object identification features corresponding to the target object include: determining the target object indicated in the video frame; detecting the display ratio of the target object in the video frame to obtain scale information; acquiring style information corresponding to the target object; and combining the scale information and the style information to obtain object identification features corresponding to the target object.

[0139] Optionally, in some possible implementations, obtaining the style information corresponding to the target object includes: determining a learnable vector corresponding to the target object, wherein a weight parameter of the learnable vector is adjusted based on a training process of a diffusion probability model; and configuring corresponding style information based on the learnable vector.

[0140] Optionally, in some possible implementations, the object action is used to configure a virtual agent, and a method for inputting the action information includes tag setting, text input, or voice input.

[0141] By obtaining the input video for training, and performing key point detection on each video frame in the input video to obtain a key point sequence; then the key point sequence is noise configured in time order to obtain a noise sequence; and the feature information corresponding to the input video is extracted, the feature information includes the audio features corresponding to each video frame, and the semantic features associated with the audio features; and in the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform the training process, and the training process includes the first stage and the second stage, the first stage is based on the audio features, and the second stage is based on the semantic features in the latent space of the first stage; the trained diffusion probability model can be used to synthesize object actions. The diffusion probability model obtained by the above training can realize a diverse action synthesis process, because in the training process, the second stage continues to generate a stable sequence structure in the latent space of the first stage, and the semantic features are described in detail on the basis of the audio features, thereby improving the accuracy of action synthesis.

[0142] The action synthesis device provided in this embodiment belongs to the same application concept as the method provided in the above embodiments of this application, can execute the action synthesis method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the action synthesis method. For technical details not fully described in this embodiment, please refer to the specific processing content of the action synthesis method provided in the above embodiments of this application, which will not be repeated here.

[0143] The functions implemented by the above synthesis unit 701 and training unit 702 can be implemented by the same or different processors respectively, and the embodiment of the present application is not limited thereto.

[0144] It should be understood that each unit in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units above are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all units above. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0145] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0146] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0147] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0148] The embodiment of the present application also provides an action synthesis device; the action synthesis device includes a processor and an input-output component. The input-output component is used to obtain request information;

[0149] The processor is used to synthesize the action corresponding to the request information obtained by the input-output component by executing any one of the action synthesis methods in the above embodiments.

[0150] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0151] Optionally, the embodiment of the present application further provides a system, wherein the interactive client is used to obtain request information and send the request information to the server, and to display the action synthesis result output by the server;

[0152] The server is used to synthesize the action corresponding to the request information obtained by the interactive client by executing the action synthesis method described in any one of the above embodiments.

[0153] Another embodiment of the present application also provides an action synthesis device, see Figure 8 As shown, the device includes:

[0154] Memory 800 and processor 810;

[0155] The memory 800 is connected to the processor 810 and is used to store programs;

[0156] The processor 810 is used to implement the action synthesis method disclosed in any of the above embodiments by running the program stored in the memory 800.

[0157] Specifically, the above-mentioned action synthesis device may further include: a bus, a communication interface 820 , an input device 830 and an output device 840 .

[0158] The processor 810, the memory 800, the communication interface 820, the input device 830 and the output device 840 are connected to each other via a bus.

[0159] A bus may include a pathway that transfers information between components of a computer system.

[0160] The processor 810 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0161] The processor 810 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0162] The memory 800 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 800 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0163] The input device 830 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0164] Output device 840 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0165] The communication interface 820 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0166] The processor 810 executes the program stored in the memory 800 and calls other devices, which can be used to implement each step of any action synthesis method provided in the above embodiments of the present application.

[0167] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the action synthesis method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0168] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0169] In addition, the embodiment of the present application may also be a storage medium on which a computer program is stored. The computer program is executed by a processor to execute the steps of the action synthesis method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically the following steps may be implemented:

[0170] inputting the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information;

[0171] During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

[0172] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0173] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0174] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0175] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0176] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0177] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0178] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0179] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0180] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0181] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0182] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An action synthesis method, characterized in that: include: inputting the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information; During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

2. The method according to claim 1, characterized in that The training process of the diffusion probability model includes: Acquire an input video, and perform key point detection on each video frame in the input video to obtain a key point sequence; Performing noise configuration on the key point sequence in time order to obtain a noise sequence; Extracting feature information corresponding to the input video, the feature information including audio features corresponding to each of the video frames and semantic features associated with the audio features; In the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform a training process, and the training process includes a first stage and a second stage. The first stage is performed based on the audio features, and the second stage is performed in the latent space of the first stage based on the semantic features.

3. The method according to claim 2, characterized in that In the process of denoising the noise sequence, the noise sequence and the feature information are used as conditions to guide the diffusion probability model to perform a training process, including: In the process of denoising the noise sequence, the noise sequence and the feature information are input into the diffusion probability model, the diffusion probability model includes at least one first diffusion module and at least one second diffusion module, and the first diffusion module and the second diffusion module have a corresponding relationship; Executing the first stage of training process based on the first diffusion module to obtain first loss information; Inputting the output features of each of the first diffusion modules into the corresponding second diffusion module, so that the second diffusion module uses the output features as conditions to perform the second stage training process based on the semantic features to obtain second loss information; The diffusion probability model is trained according to the first loss information and the second loss information.

4. The method according to claim 3, characterized in that The performing the first stage of training process based on the first diffusion module to obtain first loss information includes: splicing the audio feature, the time feature and the object identification feature in the feature information to obtain a spliced ​​feature; Transforming the concatenated features into a latent space through a linear layer to obtain latent space features; The latent space features are input into the first diffusion module, so that the first diffusion module performs the first stage training process based on the self-attention mechanism to obtain the first loss information.

5. The method according to claim 3, characterized in that: The step of inputting the noise sequence and the characteristic information into the diffusion probability model comprises: Acquire batch data corresponding to a denoising process, the batch data comprising a guide frame and a training frame, the guide frame being determined based on a plurality of adjacent preceding training frames; The noise sequence and the feature information corresponding to the batch data are input into the diffusion probability model.

6. The method according to claim 5, characterized in that Get the batch data corresponding to the denoising process, including: If the batch data is the first batch, determining the number of frames corresponding to the guide frame; Superimposing the number of frames corresponding to the guide frame and the number of frames corresponding to the training frame to obtain a target number; Frame extraction is performed during denoising based on the target quantity to obtain the batch data.

7. The method according to claim 2, characterized in that The extracting feature information corresponding to the input video includes: Determine a video frame corresponding to the input video to extract audio data corresponding to the video frame; Extracting features from the audio data based on preset dimensions to obtain audio features, wherein the preset dimensions include rhythm, prosody and spectrum; Determining a semantic feature associated with the audio feature; Obtaining a time sequence corresponding to the video frame to obtain a time feature; Determining a target object indicated in the video frame, and extracting an object identification feature corresponding to the target object; The audio feature, the semantic feature, the time feature and the object identification feature are combined to obtain the feature information.

8. The method according to claim 7, characterized in that The determining of the semantic feature associated with the audio feature comprises: Get the preset action set; Marking the start and end time in the video frame based on the actions in the preset action set; Configure the position code according to the start and end times; A semantic feature associated with the audio feature is obtained based on the position code and the corresponding action.

9. The method according to claim 7, characterized in that: The determining the target object indicated in the video frame and extracting the object identification feature corresponding to the target object includes: determining a target object indicated in the video frame; Detecting the display ratio of the target object in the video frame to obtain scale information; Obtaining style information corresponding to the target object; The scale information and the style information are combined to obtain an object identification feature corresponding to the target object.

10. The method according to claim 9, characterized in that The acquiring the style information corresponding to the target object includes: Determining a learnable vector corresponding to the target object, wherein a weight parameter of the learnable vector is adjusted based on a training process of a diffusion probability model; Corresponding style information is configured based on the learnable vector.

11. The method according to claim 1, characterized in that: The object action is used to configure the virtual agent, and the input method of the action information includes tag setting, text input or voice input.

12. An action synthesis device, characterized in that: include: a synthesis unit, configured to input the action information into a diffusion probability model so that the diffusion probability model synthesizes the object action based on the action information; During the training process of the diffusion probability model, when denoising the noise sequence corresponding to the input video, the noise sequence and the feature information corresponding to the input video are used as conditions to guide the diffusion probability model to execute the training process, the feature information includes the audio features corresponding to each video frame of the input video, and the semantic features associated with the audio features. The training process includes a first stage and a second stage, the first stage is performed based on the audio features, and the second stage is performed based on the semantic features in the latent space of the first stage.

13. An action synthesis device, comprising an input and output component and a processor; The input and output components are used to obtain request information; The processor is used to synthesize the action corresponding to the request information obtained by the input and output component by executing the action synthesis method as described in any one of claims 1 to 11.

14. An action synthesis system, comprising an interactive client and a server; The interactive client is used to obtain request information and send the request information to the server, and to display the action synthesis result output by the server; The server is used to synthesize the action corresponding to the request information obtained by the interactive client by executing the action synthesis method described in any one of claims 1 to 11.

15. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the action synthesis method according to any one of claims 1 to 11 is implemented.

Images