Video generation method, deep learning model training method, device, equipment and storage medium

Abstract

The present disclosure provides a video generation method, which relates to the field of artificial intelligence technology, in particular to the field of deep learning and computer vision technology, and can be applied to artificial intelligence video generation scenarios. The specific implementation scheme is: based on image features, text features, and action features of the target object in the image, generate an action sequence of the target object; determine the motion control features of the action sequence, the motion control features are used to control the action of the target object in the generated video to be consistent with the action sequence; and generate a target video based on image features, text features and motion control features. The present disclosure also provides a training method, device, electronic device and storage medium for a deep learning model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning and computer vision technology, and can be applied to artificial intelligence video generation scenarios. More specifically, the present disclosure provides a video generation method, a training method for a deep learning model, a device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] With the continuous development of artificial intelligence technology, artificial intelligence content generation has received widespread attention. Among them, artificial intelligence video generation refers to generating a video with a certain sense of reality based on various types of user input (such as images, text, etc.), while maintaining the continuity, picture clarity, and rationality of video frames. Summary of the invention

[0003] The present disclosure provides a video generation method, a deep learning model training method, a device, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, a video generation method is provided, the method comprising: generating an action sequence of a target object based on image features, text features, and action features of the target object in an image; determining motion control features of the action sequence, the motion control features being used to control the action of the target object in the generated video to be consistent with the action sequence; and generating a target video based on the image features, text features, and motion control features.

[0005] According to a second aspect, a training method for a deep learning model is provided, the method comprising: determining a sample action sequence of a sample object in a sample image sequence; inputting a starting sample image in the sample image sequence, a starting sample action in the sample action sequence, and a sample text into the deep learning model to obtain an output action sequence; determining the loss of the deep learning model based on the difference between the sample action sequence and the output action sequence; and adjusting the parameters of the deep learning model based on the loss.

[0006] According to a third aspect, a training method for a deep learning model is provided, the method comprising: determining a sample action sequence of a sample object in a sample image sequence; determining image sequence features of the sample image sequence, and denoising the image sequence features to obtain noisy image sequence features; inputting a starting sample image, a sample action sequence, a noisy image sequence features, and a sample text in the sample image sequence into a deep learning model to obtain denoised image sequence features; determining an output image sequence based on the denoised image sequence features; determining a loss of the deep learning model based on a difference between the sample image sequence and the output image sequence; and adjusting parameters of the deep learning model based on the loss.

[0007] According to a fourth aspect, a video generating device is provided, which includes: an action sequence generating module, which is used to generate an action sequence of a target object based on image features, text features, and action features of the target object in an image; a motion control feature determining module, which is used to determine the motion control features of the action sequence, and the motion control features are used to control the action of the target object in the generated video to be consistent with the action sequence; and a video generating module, which is used to generate a target video based on image features, text features, and motion control features.

[0008] According to a fifth aspect, a training device for a deep learning model is provided, the device comprising: a first sample determination module, used to determine a sample action sequence of a sample object in a sample image sequence; an action sequence output module, used to input a starting sample image in the sample image sequence, a starting sample action in the sample action sequence, and a sample text into the deep learning model to obtain an output action sequence; a first loss determination module, used to determine the loss of the deep learning model based on the difference between the sample action sequence and the output action sequence; and a first adjustment module, used to adjust the parameters of the deep learning model based on the loss.

[0009] According to a sixth aspect, a training device for a deep learning model is provided, the device comprising: a second sample determination module for determining a sample action sequence of a sample object in a sample image sequence; a denoising module for determining an image sequence feature of the sample image sequence, and denoising the image sequence feature to obtain a noisy image sequence feature; a denoising module for inputting a starting sample image, a sample action sequence, a noisy image sequence feature and a sample text in the sample image sequence into a deep learning model to obtain a denoised image sequence feature; an image sequence output module for determining an output image sequence based on the denoised image sequence feature; a second loss determination module for determining the loss of the deep learning model based on the difference between the sample image sequence and the output image sequence; and a second adjustment module for adjusting the parameters of the deep learning model based on the loss.

[0010] According to the seventh aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0012] According to a ninth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is a schematic diagram of an exemplary system architecture to which a video generation method and a deep learning model training method can be applied according to an embodiment of the present disclosure;

[0016] Figure 2 is a flowchart of a video generation method according to an embodiment of the present disclosure;

[0017] Figure 3 is a schematic diagram of a video generation method according to an embodiment of the present disclosure;

[0018] Figure 4 is a schematic diagram of an action sequence generation method according to an embodiment of the present disclosure;

[0019] Figure 5 is a schematic diagram of a video generation method according to an embodiment of the present disclosure;

[0020] Figure 6 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure;

[0021] Figure 7 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure;

[0022] Figure 8 is a block diagram of a video generating apparatus according to an embodiment of the present disclosure;

[0023] Fig. 9 is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0024] Fig.10 is a block diagram of a training apparatus for a deep learning model according to another embodiment of the present disclosure;

[0025] Fig.11It is a block diagram of an electronic device according to at least one of a video generation method and a deep learning model training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Traditional video generation methods include video generation based on the GAN (Generative Adversarial Network) model and video generation based on 3D reconstruction and rendering. Among them, videos generated based on the GAN model are usually of low quality, with poor image stability and easy shaking. Video generation based on 3D reconstruction and rendering is costly and cumbersome, and can generally only generate multi-view static videos.

[0028] The method of driving video generation based on guidance signals is gaining more and more attention. Guidance signals can include images, text, audio, etc. This generation method can be widely used in scenarios such as interesting video generation, image-to-animation, and film and television drama production.

[0029] For example, the generation of videos driven by guidance signals includes the generation of text-driven human action videos. Videos generated by text-driven human action need to maintain the continuity between video frames and the clarity of the picture, while also ensuring the reasonableness of the movement and consistency with the input conditions. The generation of text-driven human action videos can be used in scenarios such as short play production, dynamic advertising generation, dynamic human portrait generation, and emoticon package generation.

[0030] However, current video generation models generally generate videos directly based on input information such as text and images. Model training is difficult and the quality of the generated videos is difficult to meet requirements.

[0031] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0033] Figure 1is a schematic diagram of an exemplary system architecture to which a video generation method and a deep learning model training method can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0035] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0036] At least one of the video generation methods provided in the embodiments of the present disclosure may generally be executed by the terminal devices 101, 102, and 103. Accordingly, the video generation apparatus provided in the embodiments of the present disclosure may generally be disposed in the terminal devices 101, 102, and 103.

[0037] The training method of the deep learning model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can generally be set in the server 105.

[0038] Figure 2 is a flowchart of a video generation method according to an embodiment of the present disclosure.

[0039] like Figure 2 As shown, the video generating method 200 includes operations S210 to S230.

[0040] In operation S210, an action sequence of the target object is generated based on the image features, the text features, and the action features of the target object in the image.

[0041] For example, the image feature may be obtained by encoding the target image through an image encoder, and the text feature may be obtained by encoding the target text through a text encoder. The target image and the target text may be guide signals for driving the generation of an action sequence. The action in the action sequence may include multiple key points representing the posture of the target object. For example, the action sequence includes multiple frames of action, and each frame of action includes multiple key points representing the posture of the frame of action.

[0042] The image encoder and the text encoder may be pre-trained models, for example, both may be CLIP (Contrastive Language-Image Pre-Training, a pre-trained neural network model for matching images and texts).

[0043] The target object in the target image can be a human body, an animal, a virtual image, etc. The target text can be a text used to describe the motion information of the target object. For example, the target text includes "running", "walking", "waving", etc. The target image and the target text serve as guidance signals, and their purpose is to guide the generation of an action sequence that conforms to the motion posture of the target object and the motion description information of the target object.

[0044] By extracting the posture information of the target action from the target image, the action feature of the target object in the target image can be obtained. For example, the action feature includes multiple key points that characterize the action posture of the target object in the target image. The action feature can be used as an input action, together with the image feature and the text feature as input conditions to drive the generation of an action sequence.

[0045] Each frame of the generated action sequence includes multiple key points, which can represent the action posture of the target object. In other words, multiple key points form a frame of the target object, and multiple frames of action represent the change of the target object's action, that is, movement.

[0046] In operation S220 , motion control characteristics of the action sequence are determined.

[0047] For example, motion control features are used to control the motion of a target object in the generated video to be consistent with the motion sequence.

[0048] For example, an action sequence represents the action changes of a target object. The action sequence is used together with a target image and a target text as a guiding signal to guide the generation of a video. The action sequence can make the action of the target object in the generated video more accurate.

[0049] For example, the features of the action sequence can be extracted to obtain motion control features, which can be used to control the action changes of the target object in the generated video to be consistent with the action changes in the action sequence, thereby more accurately controlling the generation of the action of the target object in the target video.

[0050] In operation S230, a target video is generated based on the image feature, the text feature, and the motion control feature.

[0051] For example, image features, text features, and motion control features may be used as input conditions to drive the generation of a target video that includes the motion changes of a target object.

[0052] According to an embodiment of the present disclosure, operation S230 includes iteratively denoising the initial noise based on image features, text features, and motion control features to obtain video features, wherein the type and duration of the initial noise are consistent with the target video; and decoding the video features to obtain the target video.

[0053] For example, a diffusion model can be used to generate videos. The diffusion model is a generative model. The training process includes gradually adding noise to the sample video until the video becomes completely noisy, and then iteratively denoising the noise to restore it to the original video. The process of adding noise is called diffusion, and the process of denoising is called inverse diffusion. Videos can be generated using the inverse diffusion process of the diffusion model.

[0054] For example, the initial noise can be determined, and the type of the initial noise being consistent with the target video can mean that the noise signal is also in a video format, such as a noisy video. The duration of the initial noise can be consistent with the duration of the video to be generated. The initial noise can be Gaussian noise. The initial noise is used as the input of the diffusion model, and the inverse diffusion process of the diffusion model is used to iteratively denoise the initial noise based on the input conditions of image features, text features, and motion control features to obtain video features. Next, the video features can be decoded using a video codec to obtain the target video. The video codec includes, for example, a video variational codec VAE (Variational Auto-Encoder).

[0055] Since it is difficult to obtain a clear video through one or a few denoising operations, the denoising operation can be performed iteratively. After multiple iterations, the noise is gradually processed into a clear video. Iterative denoising means that the output of the previous denoising operation is used as the input of the next denoising operation, and this cycle is repeated until a clear video is obtained.

[0056] The embodiments of the present disclosure guide the generation of an action sequence of a target object based on text and images, and guide the generation of a target video based on text, images and action sequences. Since the action sequence characterizes the action changes of the target object and is used to control the actions of the target object in the generated video to be consistent with the actions in the action sequence, high-quality videos can be generated.

[0057] In one example, a video can be generated based on text and images, but it is difficult to achieve precise control of actions in the video generated by this generation method. Compared with the method of directly driving and generating videos based on text and images, the embodiment of the present disclosure can improve the quality and effect of video generation by first generating an action sequence based on text and images, and then precisely controlling the action changes of the target object based on the action sequence.

[0058] Figure 3 is a schematic diagram of a video generation method according to an embodiment of the present disclosure.

[0059] like Figure 3 As shown, the video generation method includes two stages. The first stage is to generate an action sequence 322 through an action sequence generation model 310. The second stage is to generate a target video 330 through a video generation model 320.

[0060] Image 311 includes a target object. The posture information of the target object is extracted from image 311 to obtain action 312. Action 312 includes multiple key points representing the posture information of the target object. Text 313 includes motion description information of the target object, such as "running". Image 311, action 312 and text 313 are input into action sequence generation model 310 together to obtain action sequence 322.

[0061] Image 321 may be the same image as image 311 and also include the target object. In addition, image 321 may also be a different image from image 311. For example, image 321 may include background, style, scene, and other information in addition to the target object. Similarly, text 323 may be the same text as text 313 and include motion description information of the target object, such as "running". Text 323 may also be a different text from text 313 and may include background, style, scene, and other information in addition to the motion description information of the target object, such as "running on the grass".

[0062] The image 321, the action sequence 322, and the text 323 are input into the video generation model 320 to obtain the target video 330. The motion of the target object in the target video 330 is consistent with the action sequence 322, and is consistent with the motion information, style, scene and other information described in the text 323.

[0063] According to the embodiments of the present disclosure, in the first stage, an action sequence of a target object is generated based on text and image guidance, and in the second stage, the movement of the target object in a generated video is driven by the action sequence, image and text together. Compared with directly driving video generation based on images and text, the drive is simpler and more precise, which reduces the difficulty of video generation and improves video quality.

[0064] In addition, in one example, in the action sequence generation stage, the action sequence can be driven to be generated based on simple images (which can only contain target objects) and simple texts (which can only contain actions). In the video generation stage, complex images (such as images containing style, scene, etc. information) and complex texts (text containing style, scene, etc. descriptive information) can be used together with action sequences to drive the generation of richer video content and improve the video generation effect.

[0065] Combine the following Figures 4 and 5 The action sequence generation and video generation provided by the embodiments of the present disclosure are explained.

[0066] Figure 4 is a schematic diagram of an action sequence generation method according to an embodiment of the present disclosure.

[0067] According to an embodiment of the present disclosure, generating an action sequence includes determining image features and action features from a target image containing motion posture information of a target object; determining text features from a target text containing motion description information of the target object; determining position features for indicating the position order of multiple actions in the action sequence; and generating an action sequence based on the image features, text features, action features and position features.

[0068] like Figure 4 As shown, an action sequence can be generated using an action sequence generation model 410. The action sequence generation model can include an image encoder, a text encoder, and an action sequence generation network. The image encoder and the text encoder can be CLIP models. The action sequence generation network can be ViT-Unet (Vision Transformer-Unet).

[0069] Image 401 includes a target object, for example, the target object is a person, and the action of the person is running. Image 401 is input into an image encoder to obtain image features. Posture 402 of the target object is extracted from image 401 to obtain action features. For example, posture 402 includes multiple key points of a human body, which constitute the posture of a human body running. Posture 402 can be the starting posture of the target object's movement. Text 403 includes information describing the motion state of the target object, such as "running". Text 403 is input into a text encoder to obtain text features. Position information 404 indicates the position order of multiple actions in the action sequence, and the position information is encoded to obtain position features.

[0070] Image features, action features, text features, and position features are input into the action sequence generation network to obtain the action sequence X0, X1, X2, ... X n , n is, for example, an integer greater than 2.

[0071] Posture 402 is the input action X0 of the action sequence generation model, and is also the starting action in the action sequence output by the action sequence generation model 410. In other words, the first frame action output by the action sequence generation model 410 is consistent with the input action.

[0072] According to the embodiments of the present disclosure, the generation of an action sequence is driven based on text, images, initial posture, and position information representing the action sequence, which can make the generation of the action sequence more accurate.

[0073] Figure 5 is a schematic diagram of a video generation method according to an embodiment of the present disclosure.

[0074] According to an embodiment of the present disclosure, generating a video includes performing spatial cross-attention processing on image features to obtain image attention features; performing spatial cross-attention processing and temporal attention processing on text features to obtain text attention features; performing spatial cross-attention processing and temporal attention processing on motion control features to obtain motion control attention features; and iteratively denoising the initial noise based on the image attention features, the text attention features, and the motion control attention features to obtain video features. The video features are decoded to obtain the target video.

[0075] like Figure 5 As shown, a target video 504 can be generated using a video generation model 510. The video generation model 510 includes an image encoder, a text encoder, a motion control network, and a denoising network. The denoising network can be a Unet network, and the Unet network includes multiple modules, each of which includes a spatial cross-attention network 5101 and a temporal attention network 5102.

[0076] The action sequence 501 may be generated by an action sequence generation model, and each frame of motion in the action sequence 501 includes a plurality of key points representing the posture information of the target object. The image 502 includes the target object, and the posture of the target object in the image 502 is consistent with the posture of the target object in the first frame of the action in the action sequence 501. The text 503 includes the motion description information of the target object, for example, the text content includes "a boy waving with a smile".

[0077] The action sequence 501 is input into the motion control network to obtain the motion control features. The motion control features are input into the denoising network, and can act on the spatial cross-attention network 5101 and the temporal attention network 5102 in each module of the denoising network to obtain the motion control attention features. In addition to controlling the timing, the motion control features also control the action picture, so spatial cross-attention processing and temporal attention processing are required.

[0078] Image 502 is input into the image encoder to obtain image features. The image features are input into the denoising network, which can only act on the spatial cross-attention network in each module to obtain image attention features. This is because the image is more used to guide the generation of the video screen, rather than the time sequence. The spatial cross-attention network pays more attention to the spatial position relationship between pixels. By learning the position relationship between pixels, the screen can be generated well.

[0079] The text 503 is input into the text encoder to obtain text features. The text features are input into the denoising network, and in addition to acting on the spatial cross-attention network, they are also acted on the temporal attention network to obtain text attention features. This is because text is more used to guide the changes in the temporal sequence of the action in the movement, so the text features are also acted on the temporal attention network to improve the model's ability to guide the movement through text.

[0080] The denoising network iteratively denoises the initial noise based on the image attention features, the text attention features and the motion control attention features to obtain the video features. The video features are decoded using a video codec to obtain the target video 504.

[0081] According to the embodiments of the present disclosure, the target video is driven to be generated according to the text, image, and action sequence, and the action changes of the target object in the target video can be accurately controlled to improve the video quality and effect.

[0082] Combine the following Figure 6 The training method of the action sequence generation model provided in the embodiment of the present disclosure is described.

[0083] Figure 6 It is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.

[0084] like Figure 6 As shown, the deep learning model can be a model for the above-mentioned action sequence generation. The training method of the deep learning model includes operations S610 to S640.

[0085] In operation S610, a sample action sequence of a sample object in a sample image sequence is determined.

[0086] The sample image sequence may be a sample video. The sample action sequence may be obtained by extracting the posture information of the sample object in each sample image in the sample image sequence. The sample object may be a person, an action or a virtual image. The sample action sequence includes a plurality of sample actions, and each sample action includes a plurality of key points representing the posture of the sample object.

[0087] In addition, sample text may also be determined. For example, the sample text may include text content for describing the motion information of the sample object. For example, the sample text includes content such as "running", "walking", and "waving".

[0088] In operation S620, a starting sample image in the sample image sequence, a starting sample action in the sample action sequence, and a sample text are input into a deep learning model to obtain an output action sequence.

[0089] The starting sample image in the sample image sequence, the starting sample action in the sample action sequence, and the sample text can be used as guiding signals and input into the deep learning model, so that the deep learning model outputs an output action sequence that conforms to the motion posture of the sample object and the motion description information of the sample object.

[0090] For example, the first frame of action in the output action sequence output by the deep learning model is consistent with the starting sample action. The number of action frames in the sample action sequence is the same as the number of action frames in the output action sequence.

[0091] In operation S630, a loss of the deep learning model is determined according to a difference between the sample action sequence and the output action sequence.

[0092] In operation S640, parameters of the deep learning model are adjusted according to the loss.

[0093] For example, for each sample action of a frame in the sample action sequence, the difference between the sample action and the output action of the corresponding frame in the output action sequence can be calculated as the loss of the frame. The difference can be mean square error, cross entropy, etc.

[0094] The sum of the losses of all frames in the sample action sequence can be used as the loss of the deep learning model. According to the loss, the parameters of the deep learning model can be adjusted.

[0095] The embodiments of the present disclosure train a deep learning model to generate an action sequence by utilizing sample text, a starting image in a sample image sequence, and a starting action of a sample object in the starting image. Compared with the related art of training a model by only utilizing images or text to drive the generation of an action sequence, the embodiments of the present disclosure can improve the effect of a deep learning model in generating an action sequence.

[0096] According to an embodiment of the present disclosure, a deep learning model includes a text encoder, an image encoder, and an action sequence generation network. The deep learning model outputs an action sequence including inputting a starting sample image into the image encoder to obtain image features; inputting a sample text into the text encoder to obtain text features; and inputting image features, text features, and a starting sample action into the action sequence generation network to obtain an output action sequence.

[0097] The text encoder and the image encoder can be trained CLIP networks. The action sequence generation network can be a ViT-Unet network. During training, the text encoder and the image encoder can be fixed, and only the action sequence generation network needs to be trained. In other words, according to the loss of the model, only the parameters of the action sequence generation network can be adjusted.

[0098] Combine the following Figure 7 The training method of the action sequence generation model provided in the embodiment of the present disclosure is described.

[0099] Figure 7 is a flowchart of a method for training a deep learning model according to another embodiment of the present disclosure.

[0100] like Figure 7 As shown, the deep learning model can be the above-mentioned video generation model. The training method of the video generation model includes operations S710 to S760.

[0101] In operation S710, a sample action sequence of a sample object in a sample image sequence is determined.

[0102] The sample image sequence may be a sample video. The sample action sequence may be generated by the above-mentioned action sequence generation model, or may be obtained by extracting the posture information of the sample object in each sample image in the sample image sequence. The sample object may be a person, an action or an avatar. The sample action sequence includes a plurality of sample actions, and each sample action includes a plurality of key points representing the posture of the sample object.

[0103] In addition, sample text may also be determined, and the sample text may include text content for describing the motion information of the sample object, such as "running", "walking", "waving" and the like.

[0104] In addition, the sample text may also include richer content, such as picture background, picture style, picture scene, etc. For example, the sample text is "a boy is running happily on the green grass".

[0105] In operation S720, an image sequence feature of the sample image sequence is determined, and noise is added to the image sequence feature to obtain a noisy image sequence feature.

[0106] The sample image sequence may be encoded using a video codec to obtain image sequence features. The video codec may be a video variational codec VAE.

[0107] The sample noise is determined, and the sample noise may be consistent with the type of the sample image sequence, for example, the sample noise is in a video format, and the sample noise may be the same as the duration of the sample image sequence.

[0108] The image sequence features and the sample noise can be fused in a certain ratio to obtain the image sequence features with noise. For example, when the image sequence features and the sample noise are fused in a ratio of 0:1, only noise is obtained.

[0109] In one example, the image sequence features may be gradually noised until the image sequence features completely become noise.

[0110] In operation S730, the starting sample image, the sample action sequence, the noisy image sequence features, and the sample text in the sample image sequence are input into the deep learning model to obtain the denoised image sequence features.

[0111] The starting sample image, sample action sequence, and sample text in the sample image sequence can be used as guiding signals and input into the deep learning model. The deep learning model calculates the noise and denoises the noisy image sequence features based on the calculated noise to obtain the denoised image sequence features.

[0112] Operation S730 may be performed iteratively. Since it is difficult to obtain image sequence features of a real video through one or a few denoising operations, iterative denoising may be performed to generate image sequence features of a real video as much as possible.

[0113] In operation S740, an output image sequence is determined according to the denoised image sequence characteristics.

[0114] A video codec may be used to decode the denoised image sequence features to obtain an output image sequence. The video codec may be a video variational codec VAE.

[0115] In operation S750, a loss of the deep learning model is determined according to a difference between the sample image sequence and the output image sequence.

[0116] In operation S760, parameters of the deep learning model are adjusted according to the loss.

[0117] For example, for each sample image in the sample image sequence, the difference between the sample image and the output image of the corresponding frame in the output image sequence can be calculated as the loss of the frame. The difference can be mean square error, cross entropy, etc.

[0118] The sum of the losses of all frames in the sample image sequence can be used as the loss of the deep learning model. According to the loss, the parameters of the deep learning model can be adjusted.

[0119] The embodiments of the present disclosure train a deep learning model to generate videos by utilizing sample text, a starting image in a sample image sequence, and a sample action sequence in a sample image sequence. Compared with the related art of training a model by only utilizing images or text to drive the generation of videos, the embodiments of the present disclosure can improve the effect of generating videos using deep learning models.

[0120] According to an embodiment of the present disclosure, the deep learning model includes a text encoder, an image encoder, a motion control network and a denoising network. Operation S730 includes inputting a starting sample image into the image encoder to obtain image features; inputting a sample text into the text encoder to obtain text features; inputting a sample action sequence into the motion control network to obtain motion control features; and inputting the motion control features, the image features of the starting sample image, the text features of the sample text, and the noisy image sequence features into the denoising network to obtain denoised image sequence features.

[0121] The text encoder and the image encoder can be trained CLIP networks. The motion control network can be a convolutional neural network. The denoising network can be a UNet network. During training, the text encoder and the image encoder can be fixed, and only the motion control network and the denoising network need to be trained. In other words, according to the loss of the model, only the parameters of the motion control network and the denoising network can be adjusted.

[0122] In the denoising network, the noisy image sequence features can be iteratively denoised based on the motion control features, the image features of the starting sample image, and the text features to obtain the denoised image sequence features.

[0123] According to an embodiment of the present disclosure, the denoising network includes a spatial cross-attention subnetwork and a temporal attention subnetwork. The denoising network iteratively denoises the noisy image sequence features based on motion control features, image features of the starting sample image, and text features, and the denoised image sequence features are obtained by inputting the image features into the spatial cross-attention subnetwork to obtain image attention features; inputting the text features into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain text attention features; inputting the motion control features into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain motion control attention features; and iteratively denoising the noisy image sequence features based on the image attention features, the text attention features, and the motion control attention features to obtain denoised image sequence features.

[0124] For example, the image feature input denoising network can only act on the spatial cross-attention network in each module to obtain the image attention feature. Since the image is more used to guide the generation of the video picture, rather than the time sequence. The spatial cross-attention network pays more attention to the spatial position relationship between pixels. By learning the position relationship between pixels, the picture can be generated well.

[0125] For example, the text feature is input into the denoising network, and in addition to acting on the spatial cross-attention network, it also acts on the temporal attention network to obtain the text attention feature. This is because the text is more used to guide the changes in the temporal sequence of the action in the movement, so the text feature also acts on the temporal attention network to improve the model's ability to guide the movement through the text.

[0126] For example, the motion control feature is input into the denoising network, and can act on the spatial cross-attention network and temporal attention network in each module of the denoising network to obtain the motion control attention feature. This is because the motion control feature controls the action picture in addition to the timing, so spatial cross-attention processing and temporal attention processing are required.

[0127] According to an embodiment of the present disclosure, the present disclosure also provides a video generating device, a training device for a deep learning model, and another training device for a deep learning model.

[0128] Figure 8 is a block diagram of a video generating apparatus according to an embodiment of the present disclosure.

[0129] like Figure 8 As shown, the video generating device 800 includes an action sequence generating module 801 , a motion control feature determining module 802 and a video generating module 803 .

[0130] The action sequence generating module 801 is used to generate an action sequence of a target object based on image features, text features, and action features of the target object in the image.

[0131] The motion control feature determination module 802 is used to determine the motion control features of the action sequence, and the motion control features are used to control the motion of the target object in the generated video to be consistent with the action sequence.

[0132] The video generation module 803 is used to generate a target video based on image features, text features and motion control features.

[0133] The action sequence generation module 801 includes an image and action feature determination unit, a text feature determination unit, a position feature determination unit, and an action sequence generation unit.

[0134] The image and motion feature determination unit is used to determine image features and motion features from a target image containing motion posture information of a target object.

[0135] The text feature determination unit is used to determine text features from the target text containing the motion description information of the target object.

[0136] The position feature determination unit is used to determine a position feature for indicating a position order of multiple actions in an action sequence.

[0137] The action sequence generation unit is used to generate an action sequence based on image features, text features, action features and position features.

[0138] The video generation module 803 includes a video feature determination unit and a decoding unit.

[0139] The video feature determination unit is used to iteratively denoise the initial noise based on the image feature, the text feature and the motion control feature to obtain the video feature, wherein the type and duration of the initial noise are consistent with the target video.

[0140] The decoding unit is used to decode the video features to obtain the target video.

[0141] The video feature determination unit includes an image feature processing subunit, a text feature processing subunit, a motion control feature processing subunit and a video feature determination subunit.

[0142] The image feature processing subunit is used to perform spatial cross-attention processing on image features to obtain image attention features.

[0143] The text feature processing subunit is used to perform spatial cross-attention processing and temporal attention processing on text features to obtain text attention features.

[0144] The motion control feature processing subunit is used to perform spatial cross-attention processing and temporal attention processing on the motion control features to obtain motion control attention features.

[0145] The video feature determination subunit is used to iteratively denoise the initial noise based on the image attention feature, the text attention feature and the motion control attention feature to obtain the video feature.

[0146] Fig. 9 is a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0147] like Fig. 9 As shown, the training device 900 of the deep learning model includes a first sample determination module 901, an action sequence output module 902, a first loss determination module 903 and a first adjustment module 904.

[0148] The first sample determination module 901 is used to determine a sample action sequence of a sample object in a sample image sequence.

[0149] The action sequence output module 902 is used to input the starting sample image in the sample image sequence, the starting sample action in the sample action sequence, and the sample text into the deep learning model to obtain an output action sequence.

[0150] The first loss determination module 903 is used to determine the loss of the deep learning model according to the difference between the sample action sequence and the output action sequence.

[0151] The first adjustment module 904 is used to adjust the parameters of the deep learning model according to the loss.

[0152] The first sample determination module 901 is used to extract the posture information of the sample object in each sample image in the sample image sequence to obtain a sample action sequence.

[0153] The deep learning model includes a text encoder, an image encoder and an action sequence generation network; the action sequence output module 902 includes a first image feature determination unit, a first text feature determination unit and an action sequence output unit.

[0154] The first image feature determination unit is used to input the starting sample image into the image encoder to obtain the image feature.

[0155] The first text feature determination unit is used to input the sample text into the text encoder to obtain text features.

[0156] The action sequence output unit is used to input image features, text features, and starting sample actions into the action sequence generation network to obtain an output action sequence.

[0157] The first adjustment module 904 is used to adjust the parameters of the action sequence generation network according to the loss.

[0158] Fig.10 is a block diagram of a training device for a deep learning model according to another embodiment of the present disclosure.

[0159] like Fig.10 As shown, the training device 1100 of the deep learning model includes a second sample determination module 1001, a noise addition module 1002, a denoising module 1003, an image sequence output module 1004, a second loss determination module 1005, and a second adjustment module 1006.

[0160] The second sample determination module 1001 determines a sample action sequence of a sample object in a sample image sequence.

[0161] The noise adding module 1002 is used to determine the image sequence features of the sample image sequence, and add noise to the image sequence features to obtain the noisy image sequence features.

[0162] The denoising module 1003 is used to input the starting sample image, the sample action sequence, the noisy image sequence features and the sample text in the sample image sequence into the deep learning model to obtain the denoised image sequence features.

[0163] The image sequence output module 1004 is used to determine the output image sequence according to the denoised image sequence characteristics.

[0164] The second loss determination module 1005 is used to determine the loss of the deep learning model according to the difference between the sample image sequence and the output image sequence.

[0165] The second adjustment module 1006 is used to adjust the parameters of the deep learning model according to the loss.

[0166] The deep learning model includes a motion control network and a denoising network; the denoising module 1003 includes a motion control feature determination unit and a denoising unit.

[0167] The motion control feature determination unit is used to input the sample action sequence into the motion control network to obtain the motion control feature.

[0168] The denoising unit is used to input the motion control features, the image features of the starting sample image, the text features of the sample text, and the noisy image sequence features into the denoising network to obtain the denoised image sequence features.

[0169] The deep learning model also includes a text encoder and an image encoder; the denoising module also includes a second image feature determination unit and a second text feature determination unit.

[0170] The second image feature determination unit is used to input the starting sample image into the image encoder to obtain the image feature.

[0171] The second text feature determination unit is used to input the sample text into the text encoder to obtain text features.

[0172] The denoising network includes a spatial cross-attention subnetwork and a temporal attention subnetwork; the denoising unit includes an image attention feature determination subunit, a text attention feature determination subunit, a motion control attention feature determination subunit and a denoising subunit.

[0173] The image attention feature determination subunit is used to input the image features into the spatial cross-attention sub-network to obtain the image attention features.

[0174] The text attention feature determination subunit is used to input text features into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain text attention features.

[0175] The motion control attention feature determination subunit is used to input the motion control features into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain the motion control attention features.

[0176] The denoising subunit is used to iteratively denoise the noisy image sequence features based on the image attention features, the text attention features and the motion control attention features to obtain the denoised image sequence features.

[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0178] Fig.11 A schematic block diagram of an example electronic device 1100 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0179] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0180] A number of components in the device 1100 are connected to the I / O interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as at least one of the video generation method and the training method of the deep learning model. For example, in some embodiments, at least one of the video generation method and the training method of the deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of at least one of the video generation method and the training method of the deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured in any other appropriate manner (for example, by means of firmware) to execute at least one of the video generation method and the deep learning model training method.

[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0187] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Using images and texts as guide signals, using an action sequence generation model to generate an action sequence of the target object based on image features, text features, and action features of the target object in the image, wherein each frame of action in the action sequence includes a plurality of key points representing the posture of the target object; Extracting features from the key points to obtain motion control features of the action sequence, wherein the motion control features are used to control the action of the target object in the generated video to be consistent with the action sequence; and The image, text, and action sequence are used as guide signals, and based on the image features, text features, and motion control features, initial noise is iteratively denoised to obtain video features, and the video features are decoded to obtain a target video.

2. The method according to claim 1, wherein: The step of generating an action sequence of the target object based on the image features, the text features, and the action features of the target object in the image comprises: Determining the image feature and the motion feature from a target image containing motion posture information of the target object; Determining the text feature from a target text containing motion description information of the target object; determining a position feature for indicating a position order of a plurality of actions in the action sequence; and The action sequence is generated based on the image features, text features, action features and the position features.

3. The method according to claim 1, wherein: The type and duration of the initial noise are consistent with those of the target video.

4. The method according to claim 3, wherein: The iterative denoising of the initial noise based on the image features, the text features and the motion control features to obtain the video features includes: Performing spatial cross-attention processing on the image features to obtain image attention features; Performing spatial cross-attention processing and temporal attention processing on the text features to obtain text attention features; Performing spatial cross-attention processing and temporal attention processing on the motion control feature to obtain a motion control attention feature; Based on the image attention feature, the text attention feature and the motion control attention feature, the initial noise is iteratively denoised to obtain the video feature.

5. The method according to claim 1, wherein: The action sequence generation model is trained by the following operations: determining a sample action sequence of a sample object in a sample image sequence; Inputting the starting sample image in the sample image sequence, the starting sample action in the sample action sequence, and the sample text into the action sequence generation model to obtain an output action sequence; determining a loss of the action sequence generation model based on a difference between the sample action sequence and the output action sequence; and According to the loss, the parameters of the action sequence generation model are adjusted.

6. The method according to claim 5, wherein: Determining a sample action sequence of a sample object in a sample image sequence comprises: The posture information of the sample object in each sample image in the sample image sequence is extracted to obtain the sample action sequence.

7. The method according to claim 5, wherein: The action sequence generation model includes a text encoder, an image encoder and an action sequence generation network; the step of inputting the starting sample image in the sample image sequence, the starting sample action in the sample action sequence, and the sample text into the action sequence generation model to obtain an output action sequence includes: Inputting the starting sample image into an image encoder to obtain image features; Inputting the sample text into a text encoder to obtain text features; and The image features, text features, and the starting sample actions are input into the action sequence generation network to obtain the output action sequence.

8. The method according to claim 7, wherein: The step of adjusting the parameters of the action sequence generation model according to the loss includes: According to the loss, the parameters of the action sequence generation network are adjusted.

9. A method for training a deep learning model, comprising: Using an action sequence generation model, taking sample text and sample image sequence as guide signals, to generate a sample action sequence of a sample object in a sample image sequence, wherein each frame action in the sample action sequence includes a plurality of key points representing the posture of the sample object; Determining image sequence features of the sample image sequence, and adding noise to the image sequence features to obtain noisy image sequence features; Inputting the starting sample image in the sample image sequence, the sample action sequence, the noisy image sequence features and the sample text into a deep learning model to obtain denoised image sequence features; Determining an output image sequence according to the denoised image sequence characteristics; determining a loss of the deep learning model based on a difference between the sample image sequence and the output image sequence; and Based on the loss, parameters of the deep learning model are adjusted.

10. The method according to claim 9, wherein: The deep learning model includes a motion control network and a denoising network; the starting sample image in the sample image sequence, the sample action sequence, the noisy image sequence features and the sample text are input into the deep learning model to obtain the denoised image sequence features, which includes: Inputting the sample motion sequence into the motion control network to obtain motion control features; and The motion control features, the image features of the starting sample image, the text features of the sample text, and the noisy image sequence features are input into the denoising network to obtain the denoised image sequence features.

11. The method according to claim 10, wherein: The deep learning model also includes a text encoder and an image encoder; the method also includes: Inputting the starting sample image into an image encoder to obtain the image features; and The sample text is input into a text encoder to obtain the text features.

12. The method according to claim 10, wherein: The denoising network includes a spatial cross-attention subnetwork and a temporal attention subnetwork; the motion control features, the image features of the starting sample image, the text features of the sample text, and the noisy image sequence features are input into the denoising network to obtain the denoised image sequence features, including: Inputting the image features into the spatial cross-attention sub-network to obtain image attention features; Inputting the text feature into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain a text attention feature; Inputting the motion control feature into the spatial cross-attention sub-network and the temporal attention sub-network to obtain a motion control attention feature; and Based on the image attention features, the text attention features and the motion control attention features, the noisy image sequence features are iteratively denoised to obtain denoised image sequence features.

13. The method according to any one of claims 10 to 12, wherein: The adjusting the parameters of the deep learning model according to the loss includes: According to the loss, parameters of the motion control network and the denoising network are adjusted.

14. A video generating device, comprising: An action sequence generation module is used to use an image and text as guide signals and use an action sequence generation model to generate an action sequence of the target object based on image features, text features, and action features of the target object in the image, wherein each frame of action in the action sequence includes a plurality of key points representing the posture of the target object; a motion control feature determination module, configured to extract features from the key points to obtain motion control features of the action sequence, wherein the motion control features are used to control the action of the target object in the generated video to be consistent with the action sequence; and The video generation module is used to use the image, text, and action sequence as guidance signals, iteratively denoise the initial noise based on the image features, text features, and motion control features to obtain video features, and decode the video features to obtain the target video.

15. The device according to claim 14, wherein: The action sequence generation module comprises: An image and motion feature determination unit, configured to determine the image feature and the motion feature from a target image containing motion posture information of the target object; A text feature determination unit, used to determine the text feature from a target text containing the motion description information of the target object; a position feature determining unit, configured to determine a position feature for indicating a position order of a plurality of actions in the action sequence; and An action sequence generating unit is used to generate the action sequence based on the image feature, the text feature, the action feature and the position feature.

16. The device according to claim 14, wherein: The type and duration of the initial noise are consistent with those of the target video.

17. The device according to claim 16, wherein: The video generation module comprises: An image feature processing unit, used for performing spatial cross-attention processing on the image features to obtain image attention features; A text feature processing unit, used for performing spatial cross-attention processing and temporal attention processing on the text feature to obtain a text attention feature; A motion control feature processing unit, used for performing spatial cross-attention processing and temporal attention processing on the motion control feature to obtain a motion control attention feature; The video feature determination subunit is used to iteratively denoise the initial noise based on the image attention feature, the text attention feature and the motion control attention feature to obtain the video feature.

18. The apparatus according to claim 14, further comprising: A first sample determination module, used to determine a sample action sequence of a sample object in a sample image sequence; An action sequence output module, used for inputting a starting sample image in the sample image sequence, a starting sample action in the sample action sequence, and a sample text into an action sequence generation model to obtain an output action sequence; A first loss determination module, configured to determine the loss of the action sequence generation model according to the difference between the sample action sequence and the output action sequence; and The first adjustment module is used to adjust the parameters of the action sequence generation model according to the loss.

19. The device according to claim 18, wherein: The first sample determination module is used to extract the posture information of the sample object in each sample image in the sample image sequence to obtain the sample action sequence.

20. The device according to claim 18, wherein The action sequence generation model includes a text encoder, an image encoder and an action sequence generation network; the action sequence output module includes: A first image feature determination unit, configured to input the starting sample image into an image encoder to obtain image features; A first text feature determination unit, configured to input the sample text into a text encoder to obtain text features; and The action sequence output unit is used to input the image features, text features, and the starting sample action into the action sequence generation network to obtain the output action sequence.

21. The device according to claim 20, wherein: The first adjustment module is used to adjust the parameters of the action sequence generation network according to the loss.

22. A training device for a deep learning model, comprising: A second sample determination module is used to use an action sequence generation model to generate a sample action sequence of a sample object in a sample image sequence using sample text and a sample image sequence as guide signals, wherein each frame action in the sample action sequence includes a plurality of key points representing the posture of the sample object; A noise adding module, used for determining the image sequence features of the sample image sequence, and adding noise to the image sequence features to obtain the noisy image sequence features; A denoising module, used for inputting the starting sample image in the sample image sequence, the sample action sequence, the noisy image sequence features and the sample text into a deep learning model to obtain denoised image sequence features; An image sequence output module, used to determine an output image sequence according to the denoised image sequence characteristics; A second loss determination module, configured to determine the loss of the deep learning model according to the difference between the sample image sequence and the output image sequence; and The second adjustment module is used to adjust the parameters of the deep learning model according to the loss.

23. The device according to claim 22, wherein: The deep learning model includes a motion control network and a denoising network; the denoising module includes: a motion control feature determination unit, configured to input the sample motion sequence into the motion control network to obtain a motion control feature; and The denoising unit is used to input the motion control feature, the image feature of the starting sample image, the text feature of the sample text, and the noisy image sequence feature into the denoising network to obtain the denoised image sequence feature.

24. The device according to claim 23, wherein: The deep learning model also includes a text encoder and an image encoder; the denoising module also includes: A second image feature determination unit, configured to input the starting sample image into an image encoder to obtain the image feature; and The second text feature determination unit is used to input the sample text into a text encoder to obtain the text feature.

25. The device according to claim 23, wherein: The denoising network includes a spatial cross-attention sub-network and a temporal attention sub-network; The denoising unit comprises: An image attention feature determination subunit, used for inputting the image feature into the spatial cross-attention subnetwork to obtain the image attention feature; A text attention feature determination subunit, used for inputting the text feature into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain a text attention feature; a motion control attention feature determination subunit, configured to input the motion control feature into the spatial cross-attention subnetwork and the temporal attention subnetwork to obtain the motion control attention feature; and The denoising subunit is used to iteratively denoise the noisy image sequence features based on the image attention features, the text attention features and the motion control attention features to obtain denoised image sequence features.

26. The device according to any one of claims 23 to 25, wherein: The second adjustment module is used to adjust parameters of the motion control network and the denoising network according to the loss.

27. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 13.

28. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 13.

29. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, implements the method according to any one of claims 1 to 13.

Images