A video generation method, apparatus, device, medium, product

Abstract

The application discloses a video generation method, device, equipment, medium and product in the technical field of data processing, which comprises the following steps: first, acquiring a reference image, a first video and at least one skeleton graph, wherein different skeleton graphs indicate different information (such as actions and / or perspectives); second, generating at least one image according to the reference image and the at least one skeleton graph, so that the at least one image can indicate the state of an object in the reference image under the information indicated by each skeleton graph, thereby making the at least one image better present the characteristics of the object; and third, generating a second video according to the at least one image and the first video, so that the second video can indicate the state of the object in the reference image under the action sequence indicated by the first video, thereby achieving the driving of the object in the reference image to make the action sequence indicated by the video.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video generation method, apparatus, device, medium, and product. Background Technology

[0002] For some scenarios, such as video try-on and film special effects, there may be the following requirements: given a reference image and a video, drive the object in the reference image to perform the action sequence indicated by the video. Summary of the Invention

[0003] To meet the above requirements, this application provides a video generation method, apparatus, device, medium, and product.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] This application provides a video generation method, the method comprising: acquiring a reference image, a first video, and at least one skeleton diagram, wherein different skeleton diagrams indicate different information, the information including action and / or viewpoint; generating at least one image based on the reference image and the at least one skeleton diagram, the at least one image indicating the state of an object in the reference image under the information indicated by each skeleton diagram; and generating a second video based on the at least one image and the first video, the second video indicating the state of an object in the reference image under an action sequence indicated by the first video.

[0006] In one possible implementation, generating at least one image based on the reference image and the at least one skeleton diagram includes: in response to detecting that the reference image does not meet a preset constraint, generating the at least one image based on the reference image and the at least one skeleton diagram, wherein each of the at least one images meets the preset constraint, the preset constraint indicating the requirements for the body posture and / or body integrity of the objects in the images used in the video generation process, and the images used indicating the appearance of the objects in the video obtained through the video generation process.

[0007] In one possible implementation, the at least one image is generated using a first model, which includes a main network and a reference network. The input data of the reference network includes the encoding result of the reference image. The main network processes the encoding result of the at least one skeleton map based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, which generates discretized features corresponding to each skeleton map. The reference network includes a second network layer corresponding to the self-attention module, which generates discretized features corresponding to the reference image. For any skeleton map in the at least one skeleton map, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to that skeleton map, the discretized features corresponding to other skeleton maps in the at least one skeleton map, and the discretized features corresponding to the reference image, to obtain the processed features corresponding to that skeleton map.

[0008] In one possible implementation, the at least one image is generated using a first model; the training process of the first model includes: acquiring at least one label image, a skeleton map of each label image, and a reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same as the object in each label image; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model to obtain the predicted image corresponding to each label image output by the first model; and updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image.

[0009] In one possible implementation, the training process is triggered according to a first probability, and the first model learns to perform image generation processing under controlled conditions through the training process.

[0010] In one possible implementation, the training process is triggered according to a second probability, and the first model learns to generate the same image under controlled conditions through the training process; each of the labeled images is the same image, and the difference between the state of the object in the same image and the state of the object in the reference image corresponding to the at least one labeled image reaches a preset condition.

[0011] In one possible implementation, the object in the reference image corresponding to the at least one label image is in a frontal view, and the object in the same image is in a rear view.

[0012] In one possible implementation, the training process is triggered according to a third probability, and the first model learns to perform image generation processing under uncontrolled conditions through the training process; obtaining at least one labeled image, the skeleton map of each labeled image, and the reference image corresponding to the at least one labeled image includes: obtaining the at least one labeled image and the skeleton map of each labeled image, wherein the objects in different labeled images are different; inputting the skeleton map of the at least one labeled image and the reference image corresponding to the at least one labeled image into the first model includes: inputting the skeleton map of the at least one labeled image into the first model, wherein the input data of the reference network in the first model is 0.

[0013] In one possible implementation, the training process is triggered according to a fourth probability, and the first model learns to generate images describing different objects under different control conditions through the training process; obtaining at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image includes: obtaining the at least one label image, the skeleton map of each label image, and the reference image matched by each label image, wherein the objects in different label images are different, and for any label image, the object in the label image and the object in the reference image matched by the label image are the same object; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model includes: inputting the skeleton map of the at least one label image and the reference image matched by the at least one label image into the first model.

[0014] In one possible implementation, the input data of the reference network in the first model includes the encoding result of the reference image matched by the at least one label image. The main network in the first model processes the encoding result of the skeleton map of the at least one label image based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, the first network layer generates discretized features corresponding to the skeleton map of each label image, and the reference network includes a second network layer corresponding to the self-attention module, the second network layer generates discretized features corresponding to the reference image matched by each label image. For any label image among the at least one label image, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to the skeleton map of the label image, the discretized features corresponding to the skeleton maps of other label images besides the label image among the at least one label image, and the discretized features corresponding to the reference image matched by the label image, to obtain the processed features corresponding to the skeleton map of the label image.

[0015] In one possible implementation, the training process further includes: obtaining a first text corresponding to the at least one labeled image and a second text corresponding to the at least one labeled image, wherein the first text and the second text describe the at least one labeled image; the input data of the first model further includes the first text and the second text, wherein the input data of the reference network in the first model includes at least the encoding result of the second text, and the input data of the main network in the first model includes at least the encoding result of the first text.

[0016] In one possible implementation, the training process further includes: adding new modules to each attention module in the first model; updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image includes: updating each of the new modules in the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image.

[0017] In one possible implementation, the second video is further generated based on the empty background image of each of the at least one image; for any of the at least one images, the process of obtaining the empty background image of the image includes: identifying a foreground region from the image, the foreground region indicating the position of an object in the image; removing the foreground region from the image; and performing image restoration processing on the image to obtain the empty background image of the image.

[0018] In one possible implementation, the second video is generated using a second model trained on a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video. The process of obtaining the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video includes: extracting a video segment from a third video to obtain the sample video, extracting a first image from the third video to obtain the reference image corresponding to the sample video, and using the empty background image of the first image as the empty background image of the reference image corresponding to the sample video.

[0019] In one possible implementation, the process of obtaining the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video includes: extracting the video segment and the first image from the third video, and extracting the second image from the fourth video, wherein there is a difference between the empty background image of the second image and the empty background image of the first image; performing background replacement processing on each image in the video segment based on the empty background image of the second image to obtain the sample video; performing background replacement processing on the first image based on the empty background image of the second image to obtain the reference image corresponding to the sample video; and using the empty background image of the second image as the empty background image of the reference image corresponding to the sample video.

[0020] In one possible implementation, the second video is generated using a second model trained on a sample video; the process of acquiring the sample video includes: extracting a video segment from a third video, wherein the object in the first frame of the video segment is in a frontal view and the object in the last frame of the video segment is in a rear view, and determining the sample video based on the video segment.

[0021] In one possible implementation, the at least one skeleton diagram includes a skeleton diagram of an image selected from the first video.

[0022] In one possible implementation, the second video is generated based on the skeleton diagrams of each image in the first video.

[0023] This application provides a video generation apparatus, comprising: a data acquisition unit for acquiring a reference image, a first video, and at least one skeleton diagram, wherein different skeleton diagrams indicate different information, including actions and / or viewpoints; a first generation unit for generating at least one image based on the reference image and the at least one skeleton diagram, wherein the at least one image indicates the state of an object in the reference image under the information indicated by each skeleton diagram; and a second generation unit for generating a second video based on the at least one image and the first video, wherein the second video indicates the state of an object in the reference image under an action sequence indicated by the first video.

[0024] This application provides an electronic device, the device comprising: a processor and a memory; the memory for storing instructions or computer programs; the processor for executing the instructions or computer programs in the memory to cause the electronic device to perform the video generation method provided in this application.

[0025] This application provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform the video generation method provided in this application.

[0026] This application provides a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the video generation method provided in this application.

[0027] Compared with related technologies, this application has at least the following advantages:

[0028] In the technical solution provided in this application, a reference image (such as an image given by the user that can indicate the appearance of an object in the final generated image), a first video (a video given by the user indicating a sequence of actions), and at least one skeleton diagram (such as a skeleton diagram describing some simple actions) are first acquired. Different skeleton diagrams indicate different information (such as actions and / or perspectives). Then, based on the reference image and the at least one skeleton diagram, at least one image (such as some images that can present simple actions and have a complete frame) is generated so that the at least one image can indicate the state of the object in the reference image under the information indicated by each skeleton diagram, thereby enabling the at least one image to better present the characteristics of the object. Then, based on the at least one image and the first video, a second video is generated so that the second video can indicate the state of the object in the reference image under the action sequence indicated by the first video. This enables the object in the reference image to perform the action sequence indicated by the video.

[0029] As can be seen, since the above-mentioned at least one skeleton diagram can indicate some simple actions, the at least one image generated based on the at least one skeleton diagram and the reference image can more accurately and comprehensively describe the characteristics of the object in the reference image, thereby enabling the video generated based on the at least one image to better maintain the characteristics of the object, which is beneficial to improving the video generation effect. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram illustrating a video generation method provided in an embodiment of this application;

[0032] Figure 2A schematic diagram illustrating a video driven by a reference image, provided as an embodiment of this application;

[0033] Figure 3 A schematic diagram illustrating an optimization process for a reference image provided in an embodiment of this application;

[0034] Figure 4 A schematic diagram illustrating an information sharing method under a single control condition, provided as an embodiment of this application;

[0035] Figure 5 A schematic diagram illustrating an information sharing method under multiple control conditions provided in an embodiment of this application;

[0036] Figure 6 A schematic diagram of an optimization process for another reference image provided in an embodiment of this application;

[0037] Figure 7 A schematic diagram of a video driving process provided in an embodiment of this application;

[0038] Figure 8 A schematic diagram illustrating a process for obtaining an empty background image as provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the structure of a video generation device provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] Research has found that in some scenarios, such as costume changes or film and television special effects, the diffusion model can be used to achieve the processing requirement of "given a reference image and a video, drive the object in the reference image to perform the action sequence indicated by the video" (also known as video-driven requirement).

[0042] The study also found that the above-described scheme imposes very strict restrictions on the reference image, such as requiring the object in the reference image to have a simple pose and a complete body. However, in actual use, since the reference image provided by the user is obtained by shooting real-life scenes, it may present problems such as complex poses and incomplete bodies (also known as incomplete frame). As a result, the reference image cannot meet the aforementioned strict restrictions, leading to a poor quality video generated by the scheme. This results in poor robustness of the scheme.

[0043] Based on the above research, in order to overcome the above-mentioned defects, this application provides a video generation method, which includes: firstly acquiring a reference image (such as an image given by a user that does not meet the above strict limitations), a first video (a video given by a user indicating a sequence of actions), and at least one skeleton diagram (such as a skeleton diagram describing some simple actions), wherein different skeleton diagrams indicate different information (such as actions and / or perspectives); then generating at least one image (such as some images that can present simple actions and have a complete frame) based on the reference image and the at least one skeleton diagram, so that the at least one image can indicate the state of the object in the reference image under the information indicated by each skeleton diagram, thereby enabling the at least one image to better present the characteristics of the object; then generating a second video based on the at least one image and the first video, so that the second video can indicate the state of the object in the reference image under the action sequence indicated by the first video, thus enabling the object in the reference image to perform the action sequence indicated by the video. In this context, because the at least one skeleton diagram can indicate some simple actions, the at least one image generated based on the at least one skeleton diagram and the reference image can better meet the above-mentioned strict constraints, thereby enabling the at least one image to more accurately and comprehensively describe the characteristics of the object in the reference image, and thus enabling the video generated based on the at least one image to better maintain the characteristics of the object, which is beneficial to improving the video generation effect.

[0044] Furthermore, this application does not limit the entity executing the video generation method. For example, the method can be applied to a terminal device or a server. Alternatively, the method can be implemented through data interaction between the terminal device and the server. The terminal device can be a smartphone, computer, personal digital assistant (PDA), tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0046] To better understand the technical solution provided in this application, the video generation method provided in this application will be explained below with reference to some accompanying drawings. For example... Figure 1 As shown, the video generation method provided in this application includes S1-S3 below.

[0047] S1: Acquire a reference image, a first video, and at least one skeleton diagram, with different skeleton diagrams indicating different information, including action and / or viewpoint.

[0048] Here, a reference image refers to an image provided by the user that indicates the appearance of an object (e.g., ...). Figure 2 The reference image shown is used to describe the final video to be generated (e.g., the reference image). Figure 2 The appearance of the objects in each image within the video 2) shown, such as their faces, clothing, and body shape. It should be noted that this application does not limit the implementation method of the object; for example, the object can be implemented using real people, virtual images, or animals.

[0049] Furthermore, this application does not limit the method of acquiring the reference image. For example, the reference image may be provided or selected by the user through some input device. Alternatively, the reference image may be provided by other tasks within the execution device of the aforementioned video generation method.

[0050] The first video refers to a video provided by the user that indicates a sequence of actions (e.g., ...). Figure 2 The first video (as shown in video 1) is used to describe the final video to be generated (e.g., video 1). Figure 2 The actions of objects in each image within the video shown in video 2).

[0051] Furthermore, this application does not limit the method of acquiring the first video. For example, the first video may be provided or selected by the user through some input device. Alternatively, the first video may be provided by other tasks in the execution device of the aforementioned video generation method.

[0052] At least one skeleton map refers to the skeleton map that is required when optimizing a reference image (e.g., Figure 3 The at least one skeleton diagram (as shown) indicates different information (such as action and / or viewpoint) in the reference image, so that the at least one skeleton diagram can more clearly, accurately and comprehensively describe the characteristics of the object in the reference image, such as appearance, body state in different postures, etc. It should be noted that this application does not limit the implementation of this information; for example, the information may include action and / or viewpoint. The action can indicate the body posture of an object in an image. The viewpoint can indicate the shooting angle of an object in an image.

[0053] Furthermore, in some scenarios, the aforementioned at least one skeleton diagram can at least satisfy the following constraints: the actions indicated by each skeleton diagram in the at least one skeleton diagram are simple actions, and the bodies indicated by each skeleton diagram in the at least one skeleton diagram are complete, so that the reference image (e.g., ...) can be referenced based on the at least one skeleton diagram. Figure 3The image obtained by optimizing the reference image shown can not only present the simple action, but also the complete body, thus better meeting the above strict constraints.

[0054] Furthermore, this application does not limit the method of obtaining the aforementioned at least one skeleton diagram. For example, the at least one skeleton diagram may be a set of skeleton diagrams set by relevant personnel for the current application scenario, so that the actions indicated by these skeleton diagrams can meet the control condition constraints of the video generation process in the application scenario. The control condition constraints can indicate the limitations that the images used in the video generation process, which describe the appearance of objects in the final generated video, must meet, such as which simple actions the objects in the image perform, and whether the objects in the image have complete bodies.

[0055] S2: Based on a reference image and at least one skeleton diagram, generate at least one image indicating the state of an object in the reference image under the information indicated by each skeleton diagram.

[0056] Wherein, at least one of the above-mentioned images is obtained by optimizing a reference image based on at least one of the above-mentioned skeleton diagrams, so that the at least one image (e.g. Figure 2 or Figure 3 The multiple images shown can indicate the state of an object in the reference image under the information indicated by each skeleton diagram (such as action and / or viewpoint), thereby making the at least one image more able to meet the above-mentioned strict constraints than the reference image.

[0057] In addition, the number of images in the at least one image is the same as the number of images in the at least one skeleton diagram, so that the object state (such as action and / or viewpoint) presented by each image in the at least one image is consistent with the information (such as action and / or viewpoint) indicated by each image in the at least one skeleton diagram.

[0058] Therefore, in one possible implementation, if the at least one skeleton diagram includes N skeleton diagrams, then the at least one image may include N images, such that the state (e.g., action and viewpoint) of the object in the first image is consistent with the information indicated by the first skeleton diagram, the state (e.g., action and viewpoint) of the object in the second image is consistent with the information indicated by the second skeleton diagram, ... (and so on), and the state (e.g., action and viewpoint) of the object in the Nth image is consistent with the information indicated by the Nth skeleton diagram, where N is a positive integer.

[0059] Furthermore, this application does not limit the implementation of S2 described above. For example, it can utilize any pre-built machine learning model capable of generating new images based on some skeleton diagrams and reference images, such as any diffusion model or Figure 3The machine learning model based on dual UNets is implemented as shown. The UNet network is a symmetrical network exhibiting a U-shaped structure, with downsampling on the left side and upsampling on the right side.

[0060] It should be noted that, for Figure 3 The machine learning model based on dual UNets shown in this application has an image encoder that can encode an image to obtain the encoded result. Furthermore, this application does not limit the implementation of the image encoder; for example, the image encoder can be any encoder capable of encoding images, such as the encoding module in a Variational Auto-Encoder (VAE). Additionally, the reference network in this model can be derived from an image (such as...) Figure 3 The reference image shown extracts features to be used by the main network in the model; and this reference network can be implemented using a UNet network. Furthermore, the main network in the model can use the features provided by the reference network to process certain data (such as...) Figure 3 The model processes random noise and multiple skeleton maps. The main network is also implemented using the UNet network, and its network structure is the same as the reference network. However, the parameters in the main network differ from those in the reference network, so that the functions implemented by the main network (such as image generation) differ from those implemented by the reference network (such as feature extraction). Furthermore, the image decoder in this model can decode the output data of the main network to obtain new images (such as...). Figure 3 (Multiple images shown); and this application does not limit the implementation of the image decoder. For example, the image decoder can be implemented using any decoder capable of decoding image features, such as the decoding module in VAE.

[0061] It should also be noted that, for Figure 3 Regarding the dual-UNet-based machine learning model shown, this application does not limit the way the main network in the model uses the output data of the reference network in the model. For example, for any self-attention module in the main network (such as...) Figure 4 For the attention module shown, the input data of the self-attention module is obtained by applying the self-attention module to the corresponding upper network layer in the main network (such as...). Figure 4 The output data of the neural network layer 2 shown, and the network layer in the reference network that is at the same position as the previous network layer (such as...) Figure 4 The output data of the neural network layer 1 shown is obtained by splicing together the data.

[0062] It should also be noted that, for Figure 3 Regarding the two image encoders shown, they can be implemented using the same encoder or different encoders; this application does not make any specific limitation in this regard.

[0063] It should also be noted that, for Figure 3 For the random noise shown, because Figure 3 The machine learning model based on dual UNet shown is a diffusion model, meaning that the random noise in the model's input data refers to randomly generated noise data; and the number of random noise items in the model's input data is related to the number of random noise items in the input data. Figure 3 The number of images in the multiple skeleton diagrams shown is the same.

[0064] Based on the above Figure 3 As can be seen from the relevant content of the machine learning model based on dual UNet shown, in one possible implementation, when the above-mentioned at least one skeleton map includes N skeleton maps, the above-mentioned S2 may specifically include: combining the reference image, the i-th skeleton map, and randomly generated noise data (such as... Figure 3 The random noise shown is input into the model so that the model can process the noise data according to the reference image and the i-th skeleton image to generate the i-th image. The i-th image can indicate the state of the object in the reference image under the information indicated by the i-th skeleton image (such as action and viewpoint), so that the body state of the object in the i-th image (such as the posture of each body part, the orientation of the body, etc.) is consistent with the body state indicated by the i-th skeleton image, and other content in the i-th image other than the body state (such as the appearance of the object, the background, etc.) is consistent with the corresponding content in the reference image. i is a positive integer, i≤N.

[0065] S3: Based on at least one image and a first video, generate a second video that indicates the state of an object in the reference image under the action sequence indicated by the first video.

[0066] The second video is the result of driving the first video according to at least one of the above images (e.g., Figure 2 The video shown (2) is used to make the appearance of the object in the second video match that of the at least one image (such as...). Figure 2 The appearance of objects in the multiple images shown is kept consistent, and the action sequence indicated by the second video is kept consistent with the action sequence indicated by the first video, so that the second video can indicate the state of the object in the at least one image under the action sequence indicated by the first video, thereby enabling the second video to indicate the reference image (such as...). Figure 2The second video represents the state of the object in the reference image under the action sequence indicated by the first video, so that the second video can represent the result of driving the object in the reference image to perform the corresponding action in the first video.

[0067] Furthermore, this application does not limit the implementation of the background in the second video. For example, in some scenarios, such as when the background of the original video is preserved during video driving, the background in the second video can be obtained by preserving the background in the first video. As another example, in some scenarios, such as when only the action sequence in the original video is preserved during video driving without preserving other content, the background in the second video can be obtained by preserving the background in at least one of the aforementioned images.

[0068] Furthermore, this application does not limit the implementation of the above-described S3. For example, it can be implemented using any pre-built machine learning model capable of driving the first video based on some images, such as a Diffusion Transformer (DIT) model based on Transformer. Here, the DIT is a diffusion model with Transformer as its core; and the DIT is obtained by replacing the UNet network in the traditional diffusion model with a Transformer.

[0069] Based on the above content in S1 to S3, it is clear that the video driver solution provided in this application (such as...) Figure 2In the video-driven scheme shown, a reference image (such as an image given by the user that does not meet the above strict limitations), a first video (a video of a sequence of actions given by the user), and at least one skeleton diagram (such as a skeleton diagram describing some simple actions) are first acquired. Different skeleton diagrams indicate different information (such as actions and / or perspectives). Then, based on the reference image and the at least one skeleton diagram, at least one image (such as some images that can present simple actions and have a complete frame) is generated so that the at least one image can indicate the state of the object in the reference image under the information indicated by each skeleton diagram, thereby enabling the at least one image to better present the characteristics of the object. Then, based on the at least one image and the first video, a second video is generated so that the second video can indicate the state of the object in the reference image under the action sequence indicated by the first video. In this way, the object in the reference image can be driven to perform the action sequence indicated by the video. In this context, because the at least one skeleton diagram can indicate some simple actions, the at least one image generated based on the at least one skeleton diagram and the reference image can better meet the above-mentioned strict constraints, thereby enabling the at least one image to more accurately and comprehensively describe the characteristics of the object in the reference image, and thus enabling the video generated based on the at least one image to better maintain the characteristics of the object, which is beneficial to improving the video generation effect.

[0070] In addition, in some scenarios, to better improve flexibility, this application also provides an implementation of the above-mentioned at least one skeleton diagram. In this implementation, the at least one skeleton diagram includes the skeleton diagram of an image selected from the first video, so that the at least one skeleton diagram can describe some simple actions in the first video. This not only improves the flexibility of obtaining the at least one skeleton diagram, but also helps to improve the video generation effect.

[0071] As can be seen, in one possible implementation, after obtaining the first video given by the user (e.g., ... Figure 2 Following the video 1 shown, the motion complexity of each image in the first video is first analyzed; then, based on the motion complexity, some images are selected from the first video so that the motion complexity of these images does not exceed the motion complexity of other images in the video besides these images; then, based on the skeleton diagrams of these images, at least one skeleton diagram is determined so that the at least one skeleton diagram includes the skeleton diagrams of these images, so that the at least one skeleton diagram can represent some simple actions that appear in the first video.

[0072] It should be noted that, for any image in the first video, the motion complexity of that image can indicate the complexity of the object's action in the image, so that the motion complexity can represent whether the object's action in the image is a simple action; moreover, this application does not limit the method of obtaining the motion complexity of the image. For example, it can be implemented using a pre-built machine learning model with the function of evaluating the motion complexity of images. Alternatively, the motion complexity of the image can be obtained by performing motion analysis on the skeleton diagram of the image. The skeleton diagram of the image can indicate the body posture of the object in the image.

[0073] Furthermore, in some scenarios, such as those emphasizing motion retention in video-driven applications, this application also provides a possible implementation of the aforementioned second video to better improve video generation quality. In this implementation, the second video can be generated based on the skeleton diagrams of each image in the first video, ensuring that the action sequence indicated by the second video is consistent with the action sequence indicated by the skeleton diagrams of each image in the first video. Since the skeleton diagrams of each image in the first video can more accurately describe the action sequence indicated by the first video, the second video generated based on these skeleton diagrams can better maintain the action sequence, thereby improving video generation quality.

[0074] As can be seen, in one possible implementation, after obtaining the first video given by the user (e.g., ... Figure 2 Following the video shown in video 1), firstly, skeleton diagrams of each image in the first video are obtained so that these skeleton diagrams can more accurately describe the action sequence indicated by the first video; then, based on the skeleton diagrams of each image in the first video and at least one of the above-mentioned images (such as...) obtained by optimizing a reference image given by the user... Figure 2 (Multiple images shown) are used to generate a second video so that the second video can better represent the state of the object in the reference image under the action sequence indicated by the first video. This helps to improve the motion retention effect during video generation, thereby improving the video generation effect.

[0075] Research has found that a user-provided reference image may or may not meet the constraint that "the pose of the object in the image is simple and the body of the object in the image is complete".

[0076] Based on the above research, in order to further improve efficiency, this application also provides a possible implementation of the above-mentioned S2. In this implementation, S2 may specifically include: in response to detecting that the above-mentioned reference image does not meet the preset constraints (such as the above-mentioned constraint that "the pose of the object in the image is simple and the body of the object in the image is complete"), generating the above-mentioned at least one image based on the reference image and the above-mentioned at least one skeleton diagram, wherein each of the at least one image satisfies the preset constraints. The preset constraints indicate the requirements of the object in the body posture and / or body integrity of the image used in the video generation process (such as an image that exists as a control condition during video driving and can indicate the appearance of the object in the final generated video), such as requirements for simple action and body integrity. The image used indicates the appearance of the object in the video obtained through the video generation process. This ensures that optimization is only performed on the reference image that does not meet the constraints, thereby improving efficiency.

[0077] As can be seen, after obtaining the reference image provided by the user (such as...) Figure 2 After the reference image shown, it is first determined whether the reference image meets the preset constraints. If it does, it can be determined that the reference image meets the above-mentioned constraint of "simple pose of the object in the image and complete body of the object in the image". Therefore, the reference image can be directly used as the control condition to drive the first video to obtain the second video. However, if it does not meet the constraint, it can be determined that the reference image does not meet the constraint of "simple pose of the object in the image and complete body of the object in the image". Therefore, at least one image can be generated based on the reference image and the above-mentioned at least one skeleton map, so that each image in the at least one image meets the preset constraints. This makes each image in the at least one image meet the constraint of "simple pose of the object in the image and complete body of the object in the image", thereby making the video generation process based on the at least one image exhibit strong robustness, which is beneficial to improving the video generation effect.

[0078] Research has found that for some scenarios, such as those where the background in the reference image does not need to be preserved in video-driven scenarios, the constraints imposed on the reference image include: the pose of the object in the image is simple, the body of the object in the image is intact, and the background in the image is simple.

[0079] Based on the above research, in order to better improve the video generation effect, this application also provides at least one of the above-mentioned images (such as...). Figure 2 or Figure 3One possible implementation of the multiple images shown is that, in this method, the at least one image may also be generated based on a preset background image. Since the background indicated by the preset background image belongs to the simple background specified by the aforementioned constraint of "simple pose of the object in the image, complete body of the object in the image, and simple background in the image," the at least one image generated based on the preset background image can better meet this constraint. This makes the video generation process based on the at least one image more robust, thus improving the video generation effect.

[0080] As can be seen, after obtaining the reference image provided by the user (such as...) Figure 2 After the reference image shown, if it is determined that the reference image does not satisfy the background constraint and does not satisfy the foreground constraint (such as the preset constraint mentioned above), it can be determined that the reference image does not satisfy the restriction that "the pose of the object in the image is simple, the body of the object in the image is complete, and the background in the image is simple". Therefore, at least one image can be generated based on the reference image, at least one skeleton image and the preset background image, so that each image in the at least one image simultaneously satisfies the foreground constraint and the background constraint.

[0081] If it is determined that the reference image satisfies the background constraint but not the foreground constraint, it can be determined that the reference image only does not satisfy the restriction that "the pose of the object in the image is simple and the body of the object in the image is complete". Therefore, based on the reference image and the above-mentioned at least one skeleton diagram, the above-mentioned at least one image can be generated so that each image in the at least one image simultaneously satisfies the foreground constraint and the background constraint.

[0082] If it is determined that the reference image satisfies the foreground constraint but not the background constraint, it can be determined that the reference image only does not satisfy the restriction that "the background in the image is simple". Therefore, at least one of the above images can be generated based on the reference image and the preset background image so that each image in the at least one image simultaneously satisfies the foreground constraint and the background constraint.

[0083] If it is determined that the reference image satisfies both the background constraint and the foreground constraint, then it can be determined that the reference image satisfies the constraint that "the pose of the object in the image is simple, the body of the object in the image is complete, and the background in the image is simple". Therefore, the reference image can be directly used as the control condition to drive the first video to obtain the second video.

[0084] Research has found that, for at least one of the above images, if the generation processes of each image in the at least one image are independent of each other, the different generation processes experienced by different images in the at least one image will lead to some unexpected inconsistencies between different images in the at least one image, such as inconsistent appearance of objects, which will affect the final video generation effect.

[0085] Based on the above research, in order to overcome the above problems, this application also provides at least one of the above-mentioned images (such as...). Figure 2 or Figure 3 One possible implementation of the multiple images shown is in which at least one image is generated using a first model (such as...). Figure 2 The first model shown or Figure 3 The first model, generated by the dual-UNet-based machine learning model shown, includes a main network and a reference network. The input data of the reference network includes the encoding result of the aforementioned reference image. The main network processes the encoding result of at least one skeleton map based on the output data of the reference network. For any self-attention module in the main network (such as...), Figure 4 The main network includes the first network layer corresponding to the self-attention module (as shown in the diagram). Figure 4 The first network layer (layer 2) generates discretized features (such as...) corresponding to each skeleton diagram. Figure 4 Features 4, 5, and 6 are shown. The reference network includes the second network layer corresponding to the self-attention module (e.g., features 4, 5, and 6). Figure 4 The neural network layer 1 shown generates the discretized features (such as...) corresponding to the reference image. Figure 4 Feature 1, Feature 2, or Feature 3 shown); for any skeleton map in the at least one skeleton map, the concatenation result between the discretized features corresponding to the skeleton map, the discretized features corresponding to other skeleton maps in the at least one skeleton map besides the skeleton map, and the discretized features corresponding to the reference image (e.g., Figure 4 The input data of the attention module shown are subjected to self-attention processing to obtain the processed features (such as...) corresponding to the skeleton diagram. Figure 4 The processed features shown are such that information can be shared between different generation links in the main network (e.g., ...). Figure 4 The information sharing method shown can improve the consistency of the generation results obtained by these generation links, thereby helping to overcome the defects that exist when the generation processes of each image in at least one of the above images are independent of each other, and thus helping to improve the generation effect.

[0086] It should be noted that, for Figure 4 In other words, Figure 4The "attention module" mentioned refers to the first model (e.g., ...). Figure 3 Any self-attention module present in the main network of the machine learning model based on dual UNet shown; Figure 4 The "neural network layer 2" mentioned in the text refers to a neural network layer in the main network that is adjacent to the "attention module" and located before the "attention module." Furthermore, this "neural network layer 2" can process the input data (such as...) of the main network. Figure 4 Each feature in the "Features in the Main Network" section is processed (e.g., tokenized) to obtain the processing result of each feature (e.g., Figure 4 The three different features shown are feature 4, feature 5, and feature 6; Figure 4 The term "neural network layer 1" refers to a neural network layer in the reference network of the first model that provides features to the "attention module." This "neural network layer 1" occupies the same position in the reference network as "neural network layer 2" occupies in the main network. Furthermore, this "neural network layer 1" can respond to the input data (such as...) Figure 4 Each feature in the "Features in the Reference Network" shown is processed (e.g., tokenized) to obtain the processing result of each feature (e.g., Figure 4 The three identical features shown are (feature 1, feature 2, and feature 3) so that the reference network can concatenate all the features in the output data of the "neural network layer 1" and send them to the main network, so that the main network can use the features it receives from the reference network and the output data of the "neural network layer 2" to construct the input data of the "attention module".

[0087] It should also be noted that, for Figure 4 For the reference network shown, since its input data only includes the reference image, in order to better influence the processing of the main network, when the input data of the main network includes M data items (such as... Figure 3 or Figure 4 When dealing with the three data sets shown, the input data of the reference network can be copied M times, allowing the reference network to process each data set according to M generation links. This enables the j-th generation link in the reference network to subsequently provide features to the j-th generation link in the main network, where j is a positive integer, j≤M, and M is a positive integer. This ensures that the reference network and the main network have the same network structure, but different functions. Therefore, in... Figure 4 In this context, feature 1, feature 2, and feature 3 are all identical.

[0088] In addition, to further improve the generation effect, this application also provides the aforementioned first model (such as...). Figure 3 The model shown or has Figure 4 The training process of the first model (shown as an information sharing model) includes the following steps: acquiring at least one label image, the skeleton map of each label image, and a reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same as the object in each label image; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model to obtain the predicted image corresponding to each label image output by the first model; then, updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image, so that the updated first model has better generation performance, so that the subsequent steps of "acquiring at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image" and subsequent steps can be continued based on the updated first model to realize the next round of training process, and so on iteratively until the first stopping condition is reached (such as the model loss of the first model is lower than the first loss threshold, the rate of change of the model loss is lower than the first rate of change threshold, the number of updates of the first model reaches the first number threshold, etc.), and the multi-round iterative training process for the first model ends.

[0089] It should be noted that the aforementioned labeled image refers to the image used as the ground truth (GT) during the training process of the first model. This is to guide the first model to generate images as closely as possible to the labeled image based on its skeleton diagram, thereby enabling the first model to better learn how to perform image generation processing under the guidance of the labeled image. Furthermore, this application does not limit the number of images in the at least one labeled image; for example, the number of images in the at least one labeled image may be the same as the number of images in the at least one image (or the number of images in the at least one skeleton diagram).

[0090] It should also be noted that the model loss of the first model can characterize the generation performance of the first model; and the model loss is determined based on the difference between the predicted image corresponding to the at least one labeled image and the at least one labeled image. This application does not limit the calculation method of the model loss; for example, it can be implemented using any model loss, such as some loss functions used when training a diffusion model.

[0091] It should also be noted that this application does not limit the process of obtaining the at least one tag image and the reference image corresponding to the at least one tag image. For example, when the number of images in the at least one tag image is N, the acquisition process can specifically be as follows: after randomly selecting a video from the video library, firstly, randomly select N+1 images from the video; then, randomly select one image from the N+1 images as the reference image corresponding to the at least one tag image, and use the other images from the N+1 images, excluding the randomly selected image, as the at least one tag image. Since these tag images and the reference image come from the same video, the object in the reference image and the object in each tag image are the same object.

[0092] Based on the above four paragraphs, it can be seen that for the training process of the first model in the current round, a video can be randomly selected from the video library; then some images can be randomly selected from the video, and one image can be randomly selected from these images as the reference image corresponding to the at least one label image, and the remaining images in these images can be used as the at least one label image. After obtaining the skeleton map of each image in the at least one label image, the skeleton map of the at least one label image and the reference image are input into the first model, so that the first model can use the reference image as a control condition to perform image generation processing on these skeleton maps, obtain and output the predicted image corresponding to each label image, so that the difference between these predicted images and these label images can characterize the generation performance of the first model under controlled conditions. In this way, the first model can learn how to perform image generation processing under controlled conditions through the training process of the current round.

[0093] Research has found that, in order to improve the model training effect, the multi-round iterative training process for the first model should include not only at least one round of training under controlled conditions, but also at least one round of training under uncontrolled conditions, so as to ensure that the final trained first model has better performance.

[0094] Based on the above research, in one possible implementation, when the current round of training of the first model belongs to the "training process executed under conditional control", the training process can at least satisfy the following constraints: the training process is triggered according to a first probability (e.g., a probability of 0.9), the first model learns to perform image generation processing under controlled conditions through the training process, and the training process of the first model can include the following steps: obtaining at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same as the object in each label image; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model to obtain the predicted image corresponding to each label image output by the first model; then, updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image, so that the image generation performance of the first model under controlled conditions can be better optimized by means of the training process.

[0095] As can be seen, for the training process of the first model in the current round, if it is detected that the training process belongs to the training process triggered according to the first probability and executed under conditional control, then a video is randomly selected from the video library; then some images are randomly selected from the video, and one image is randomly selected from these images as the reference image corresponding to the at least one label image, and the remaining images in these images are used as the at least one label image. After obtaining the skeleton map of each image in the at least one label image, the skeleton map of the at least one label image and the reference image are input into the first model, so that the first model can use the reference image as the control condition to perform image generation processing on these skeleton maps, obtain and output the predicted image corresponding to each label image, so that the difference between these predicted images and these label images can characterize the generation performance of the first model under controlled conditions. In this way, the first model can learn how to perform image generation processing under controlled conditions through the training process of the current round.

[0096] It should be noted that this application does not limit the implementation method of the first probability described above. For example, the first probability can be set according to the actual application scenario requirements. For instance, the first probability can be 0.9.

[0097] Research has found that although training the first model is expected to generate different images from different generation links, in order to better improve the consistency of different generation links in some generated content (such as the appearance of objects), at least one round of training process that requires the generation of the same image can be added to the multi-round iterative training process of the first model under controlled conditions.

[0098] The study also found that, for the aforementioned “training process that requires the generation of the same image under controlled conditions”, the greater the difference between the image that the first model needs to generate during the training process and the reference image that serves as the control condition, the more beneficial it is to improve the consistency of different generation links in the first model in some generated content (such as the appearance of objects).

[0099] Based on the above two studies, in one possible implementation, when the current round of training of the first model belongs to the "training process that needs to generate the same image under controlled conditions", the training process can at least satisfy the following constraints: the training process is triggered according to a second probability (e.g., a probability of 0.1), the first model learns to generate the same image under controlled conditions through the training process, and the training process of the first model can include the following steps: obtaining at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same object as the object in each label image, and each label... All images are the same image, and the difference between the state of the object in the same image and the state of the object in the reference image corresponding to the at least one labeled image reaches a preset condition (such as reaching the maximum). The skeleton map of the at least one labeled image and the reference image corresponding to the at least one labeled image are input into the first model to obtain the predicted images corresponding to each labeled image output by the first model. Then, based on the difference between the predicted images corresponding to the at least one labeled image and the at least one labeled image, the first model is updated. In this way, the consistency of different generation links in the first model in some generated content (such as the appearance of objects) can be better optimized through the training process, thereby helping to improve the generation performance of the first model.

[0100] It should be noted that this application does not limit the above-mentioned preset conditions. For example, the preset conditions can be determined according to the actual application scenario. For another example, the preset conditions can indicate that the difference between the state of the object in the reference image corresponding to the at least one labeled image and the state of the object in each of the at least one labeled image is sufficiently large. Furthermore, when the reference image and these labeled images all come from the same video, the preset conditions may include: the difference between the state of the object in the reference image and the state of the object in each labeled image is not less than the difference between the state of the object in other images in the video besides the reference image and the state of the object in each labeled image.

[0101] Based on the above two paragraphs, it can be seen that for the training process of the first model in the current round, if it is detected that the training process is triggered according to the second probability, executed under controlled conditions, and requires the generation of the same image, a video can be randomly selected from the video library first; then, the two images with the greatest differences can be extracted from the video. One image is used as the reference image corresponding to the at least one labeled image, and the other image is used as each image in the at least one labeled image. This allows the first model to be trained based on the reference image and these labeled images, thereby optimizing the consistency between different generation links in the first model and thus improving the generation performance of the first model.

[0102] It should be noted that this application does not limit the implementation of the second probability, as long as it is ensured that the second probability is less than the first probability. For example, the second probability can be set according to the actual application scenario requirements. For instance, the second probability can be 0.1. Therefore, when the first probability is 0.9 and the second probability is 0.1, for the training process of the first model in the current round, the training process with a probability of 0.1 belongs to the aforementioned "training process that needs to generate the same image under controlled conditions," and with a probability of 0.8, it belongs to the training process that needs to generate different images under controlled conditions.

[0103] Furthermore, to better improve the consistency of different generation chains in the first model in some generated content (such as the appearance of objects), this application also provides a possible implementation of the training process of the first model. In this method, when the current round of training of the first model belongs to the above-mentioned "training process that needs to generate the same image under controlled conditions", the training process can at least satisfy the following constraints: the training process is triggered according to a second probability (such as a probability of 0.1), the first model learns to generate the same image under controlled conditions through the training process, and the training process of the first model may include the following steps: obtaining at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image. The object in the reference image corresponding to each labeled image is the same object as the object in each labeled image, and all labeled images are the same image. The object in the reference image corresponding to at least one labeled image is in a frontal view, and the object in the same image is in a rear view. The skeleton diagram of the at least one labeled image and the reference image corresponding to the at least one labeled image are input into the first model to obtain the predicted images corresponding to each labeled image output by the first model. Then, based on the difference between the predicted images corresponding to the at least one labeled image and the at least one labeled image, the first model is updated. In this way, the consistency of different generation links in the first model in some generated content (such as the appearance of objects) can be better optimized through this training process, thereby helping to improve the generation performance of the first model.

[0104] As can be seen, this application trains the first model to generate multiple identical images by increasing the back sampling probability (as mentioned in the second probability above), so as to ensure that there is good consistency between different generation links in the first model, which is beneficial to improving the generation performance of the first model.

[0105] Research has shown that, in order to improve the training effect of the first model, it can be trained without control conditions at a certain probability (e.g., 0.1).

[0106] The study also found that, when training the first model under uncontrolled conditions, in order to better avoid the different generation links in the first model exhibiting similar generation performance, images with as large a difference as possible can be generated by the different generation links in the first model, such as images describing different objects.

[0107] Based on the above two studies, in order to better improve the performance of the first model, this application also provides a possible implementation of the training process of the first model. In this method, when the current round of training of the first model is a training process executed under uncontrolled conditions, the training process can at least satisfy the following constraints: the training process is triggered according to a third probability (such as a probability of 0.1), the first model learns to perform image generation processing under uncontrolled conditions through the training process, and the training process of the first model can include the following steps: obtaining at least one label image and the skeleton map of each label image, with different objects in different label images; inputting the skeleton map of the at least one label image into the first model to obtain the predicted image corresponding to each label image output by the first model, where the input data of the reference network in the first model is 0; then, updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image. In this way, the training process can ensure that different generation links in the first model have unique generation performance, which is conducive to better differentiating the generation performance of each generation link in the first model, thereby improving the performance of the first model in generating different images.

[0108] Based on the above two paragraphs, it can be seen that for the training process of the first model in the current round, if it is detected that the training process is a training process triggered by the third probability and executed under uncontrolled conditions, some videos can be randomly selected from the video library so that these videos describe different objects respectively; then, an image is extracted from each video to obtain at least one label image, so that the at least one label image includes the image extracted from each video, thereby making the objects described by different images in the at least one label image different. After obtaining the skeleton map of each image in the at least one label image, the skeleton map of the at least one label image is input into the first model, so that the input data of the reference network in the first model is 0, thereby enabling the first model to perform image generation processing on these skeleton maps under uncontrolled conditions, obtain and output the predicted image corresponding to each label image, so that the difference between these predicted images and these label images can characterize the generation performance of the first model under uncontrolled conditions. In this way, the generation performance of the first model can be better optimized by means of this training process.

[0109] It should be noted that this application does not limit the implementation method of the aforementioned third probability, as long as the sum of the third probability and the aforementioned first probability does not exceed 1. For example, the third probability can be set according to the actual application scenario requirements. Alternatively, the third probability can be 0.1.

[0110] Based on the above three paragraphs, it can be seen that this application trains the first model without control conditions with a probability of 0.1, so that the input data of the reference network in the first model is 0 under the training without control conditions, and the first model can learn how to generate images extracted from different videos under the training without control conditions. This is conducive to better differentiating the generation performance of different generation links in the first model, and thus helps to enhance the performance of the first model in generating different images.

[0111] Research has shown that, in order to better differentiate the generation performance of different generation links in the first model, different control conditions can be provided for different generation links in the first model to improve the generation performance of the first model under controlled conditions.

[0112] Based on the above research, in order to better improve the generation performance of the first model, this application also provides a possible implementation of the training process of the first model. In this method, when the current round of training of the first model is a training process executed under controlled conditions and requiring the generation of different images according to different control conditions, the training process can at least satisfy the following constraints: the training process is triggered according to a fourth probability (such as a probability of 0.1), the first model learns to generate images describing different objects under different control conditions through the training process, and the training process of the first model can include the following steps: obtaining at least one label image, the skeleton map of each label image, and matching of each label image. The reference image is used, and the objects in different label images are different. For any label image, the object in the label image is the same as the object in the reference image that matches the label image. The skeleton map of the at least one label image and the reference image that matches the at least one label image are input into the first model to obtain the predicted image corresponding to each label image output by the first model. Then, the first model is updated according to the difference between the predicted image corresponding to the at least one label image and the at least one label image. In this way, the generation performance of the first model under controlled conditions can be enhanced by providing different control conditions for different generation links in the first model, thereby helping to improve the training effect of the first model.

[0113] Based on the above two paragraphs, for the training process of the first model in the current round, if it is detected that the training process is triggered by the fourth probability, executed under controlled conditions, and requires generating different images according to different control conditions, then N videos can be randomly selected from the video library first, so that the N videos describe different objects respectively; then two images can be randomly selected from the i-th video, one image as the i-th label image, and the other image as the reference image for matching the i-th label image, so that the reference image matched by the i-th label image can be used as the control condition for image generation processing in the future, so that the result of the image generation processing is as close as possible to the desired image quality. The image is close to the i-th label image, where i is a positive integer, i≤N, and N is a positive integer. After obtaining the skeleton images of each of the N label images, the skeleton images of the N label images and the reference images matched by the N label images are input into the first model. This allows the first model to generate and output the predicted images corresponding to each label image based on these reference images. This enables the generation performance of the first model under controlled conditions to be determined based on the differences between these predicted images and these label images. This training process can enhance the performance of different generation links in the first model, thereby improving the generation performance of the first model.

[0114] It should be noted that this application does not limit the implementation of the fourth probability. For example, the fourth probability can be set according to the actual application scenario requirements. For instance, the fourth probability can be 0.1. Therefore, when the first probability is 0.9 and the fourth probability is 0.1, for the training process of the first model in the current round, the training process with a probability of 0.1 is a training process executed under controlled conditions, requiring the generation of different images based on different control conditions. Furthermore, the training process with a probability of 0.8 is a training process executed under controlled conditions, requiring the generation of multiple images (such as multiple different images or multiple identical images) based on the same control condition.

[0115] Furthermore, to further improve the generation performance of the first model, this application also provides a possible implementation of the training process described above, which is executed under controlled conditions and requires the generation of different images based on different control conditions. In this implementation, the first model in the training process can at least satisfy the following constraints: the input data of the reference network in the first model includes the encoding result of the reference image matching at least one label image; the main network in the first model processes the encoding result of the skeleton map of the at least one label image based on the output data of the reference network; for any self-attention module in the main network (such as...) Figure 5 The main network includes the first network layer corresponding to the self-attention module (as shown in the diagram). Figure 5 The first network layer (layer 2) generates discretized features (such as the skeleton map of each labeled image) corresponding to the skeleton map of each labeled image. Figure 5 The reference network includes the second network layer corresponding to the self-attention module (e.g., features 4, 5, and 6 shown). Figure 5 The neural network layer 1 shown generates discretized features (such as reference images) corresponding to the matching reference images of each labeled image. Figure 5 Features 7, 8, and 9 are shown as three different features; for any label image in the at least one label image, the concatenation result between the self-attention module and the discretized features corresponding to the skeleton map of the label image, the discretized features corresponding to the skeleton maps of other label images in the at least one label image besides the label image, and the discretized features corresponding to the reference image matched by the label image (e.g., Figure 5 The input data of the attention module shown are subjected to self-attention processing to obtain the processed features (such as the skeleton map of the labeled image) corresponding to the skeleton map of the labeled image. Figure 5 The processed features shown are such that information other than control conditions can be shared between different generation links in the main network (e.g., ...). Figure 5 The information sharing method shown can be used to optimize the generation effect of these generation links, thereby helping to improve the generation effect.

[0116] It should be noted that, for Figure 5 In other words, Figure 5 Compared to Figure 4 The difference is: Figure 5 The input data for the reference network shown includes multiple reference images, enabling the reference network to target specific data types. Figure 5 The different generation links in the main network shown provide features extracted from different reference images. This enables different generation links in the main network to perform generation processing under different control conditions, thereby improving the generation performance of the first model under controlled conditions.

[0117] Research has shown that DIT outperforms the Unet network. Therefore, to further improve the generation performance of the first model, both the main network and the reference network in the first model can be replaced with DIT of the same structure. This will improve the generation performance of the replaced first model (e.g., Figure 6The model shown has better generation performance. It should be noted that the information sharing method presented in the main network of the replaced first model is the same as that presented in the main network of the original first model. Furthermore, this application does not limit the implementation of the main network and reference network of the replaced first model. For example, both the main network and reference network of the replaced first model can be implemented using N multimodal Diffusion Transformer Blocks (DITs).

[0118] It can be seen that the first model, including double DIT (such as...) Figure 6 The machine learning model based on dual DIT shown is similar to the first model mentioned above, which includes dual UNet (e.g.) Figure 3 The machine learning model based on dual UNet shown is described below. Furthermore, this "first model including dual DIT" has at least the following characteristics: A copy of the main network of the "first model including dual DIT" is used as a reference network to ensure that the network structure of the main network of the "first model including dual DIT" is consistent with the network structure of the reference network; after a module in the reference network of the "first model including dual DIT" processes the input data to obtain discretized features, these discretized features are concatenated with the discretized features output by the corresponding parallel module in the main network of the "first model including dual DIT" to obtain a concatenated result, which is then used by the corresponding self-attention module (such as...) in the main network of the "first model including dual DIT". Figure 4 The attention module shown performs self-attention processing on the splicing result to obtain the output data of the self-attention module (such as...). Figure 4 (The processed features shown). It should be noted that the position of the "certain module" in the reference network is the same as the position of the "parallel module" in the main network. The position of the "parallel module" in the main network is adjacent to the position of the "self-attention module" in the main network, and the position of the "parallel module" in the main network is earlier than the position of the "self-attention module" in the main network.

[0119] The study also found that, in order to improve the performance of the DIT, text prompts can be provided to the DIT so that it can learn from the text prompts what characteristics the data it needs to generate should have.

[0120] Based on the above research, in order to better improve the generation performance of the first model, this application also provides a possible implementation of the training process of the first model. In this method, for any round of training of the first model, the training process may further include: obtaining the first text (e.g., the text corresponding to the at least one label image) Figure 6 The text prompt word 1 shown and the second text corresponding to the at least one label image (such as...) Figure 6 The text prompt word 2 shown above describes the at least one labeled image, so that the first text and the second text can describe the image to be generated in the end. Thus, the input data of the first model also includes the first text and the second text. The input data of the reference network in the first model includes at least the encoding result of the second text, and the input data of the main network in the first model includes at least the encoding result of the first text. This enables the first model to learn from these two texts what characteristics the image to be generated has. This ensures that the first model can better learn how to generate images under the guidance of some text prompt words.

[0121] It should be noted that the first text described above can describe the data that the main network in the first model expects to generate, so that the main network can learn how to perform image generation processing under the guidance of the first text during the training process; and this application does not limit the implementation of the first text. For example, the first text can describe the image that the first model expects to generate (such as at least one labeled image mentioned above).

[0122] It should also be noted that the aforementioned second text can describe the data expected to be generated by the reference network in the first model, so that the reference network can learn how to perform feature extraction processing under the guidance of the second text during the training process. Furthermore, this application does not limit the implementation of the second text, as long as it ensures that there is no conflict between the content described by the second text and the input data of the reference network (such as the reference image corresponding to at least one labeled image). Thus, in one possible implementation, the second text can indicate what features (such as clothing, appearance, etc.) to extract from the input data of the reference network. In addition, since the second text gradually degrades during model training to prevent it from affecting the trained first model, in order to accelerate the training efficiency of the first model, the second text can be determined based on the input data of the reference network to ensure that the second text can indicate what features (such as clothing, appearance, etc.) to extract from the input data of the reference network. This allows the second text to better guide the reference network to perform rapid optimization in the early stages of model training, thereby improving training efficiency.

[0123] Research has found that because DIT belongs to a large network, the first model, which includes dual DIT (such as...), is suitable. Figure 6 The machine learning model based on dual DIT shown is relatively large, which means that training the entire first model requires a lot of resources, such as computing resources and time resources.

[0124] Based on the above research, in order to better reduce the resource overhead of model training, this application also provides a possible implementation of the training process of the first model. In this way, for any round of training of the first model, the training process may include at least the following steps: adding new modules to each attention module in the first model, and updating each new module in the first model according to the difference between the predicted image corresponding to the at least one label image and the at least one label image. This can reduce the resource overhead of model training by using the additional modules added to the first model as the trained part, thereby improving the model training effect.

[0125] As can be seen, compared to the first model mentioned above, including the dual UNet (such as...), Figure 3 The training process described in the example of the dual-UNet-based machine learning model includes the first model of the dual-DIT model (such as...). Figure 6 The training process of the machine learning model based on dual DIT shown differs in the following ways: the first model, which includes dual DIT, will train in each attention module (such as the self-attention module mentioned above or...). Figure 4 The attention module shown adds a small subnetwork as a trainable part, while the parameters in other parts (such as the other networks in the first model that includes dual DIT, except for the added subnetwork) remain frozen to ensure that the parameters in the other parts are not trained.

[0126] Research has revealed that in certain scenarios, such as preserving the background of a reference image during video-driven operations, the following requirement may exist: given a reference image and a video, drive an object in the reference image to perform the corresponding action shown in the video while preserving the background of the reference image. For ease of understanding, examples are provided below.

[0127] As an example, when the reference image given by the user is obtained through real-world shooting, the background in the reference image is relatively complex. If the user wants to generate an action in the video based on the reference image, the appearance of the object in the reference image, the background in the reference image, and the action sequence in the video are maintained when driving the video based on the reference image.

[0128] The study also found that, for the reference images shown in the two paragraphs above, when the background of the reference image is relatively complex, it is difficult to generate a complex background, which makes it difficult to maintain the complex background in the reference image.

[0129] Based on the research shown in the above three paragraphs, in order to overcome the difficulties mentioned above, this application also provides a possible implementation of the second video, in which the second video can also be based on the empty background image of each image in at least one of the above images (such as...). Figure 7 or Figure 8 The second video is generated by generating objects from these empty background images (as shown), so that the generation process of the second video is achieved by generating objects from these empty background images. This means that only objects need to be generated during the generation of the second video, without generating a background (such as a complex background). This not only overcomes the shortcomings caused by the difficulty in generating complex backgrounds (such as low video quality due to poor background preservation), but also reduces the amount of generated content to improve generation efficiency, thereby improving the generation effect. Specifically, for any image in the at least one image, the process of obtaining the empty background image of that image includes: identifying the foreground region (such as...) from the image... Figure 8 The foreground region (shown) indicates the position of the object in the image; the foreground region is removed from the image; the image is inpainted to obtain an empty background image, so that the empty background image can better describe the background information carried by the image, so that the image generated using the empty background image as the base image can better preserve the background information, thereby improving the video generation effect.

[0130] It is evident that, for some scenarios, such as those involving preserving complex backgrounds in a reference image during video-driven processing, optimizing a user-provided reference image to obtain at least one of the aforementioned images (e.g.) Figure 7 After showing multiple images, first process each image in at least one of the images (e.g., ... Figure 8 The process (as shown) is used to obtain empty background images of each image in the at least one image, so that these empty background images can better represent the background information carried by the at least one image; then, based on these empty background images, the at least one image, and a first video (such as...) given by the user... Figure 7 The skeleton diagrams of each image in video 1) are used to generate a second video (such as...). Figure 7 The video shown in video 2) enables the objects in the reference image to perform the corresponding actions in the first video to be drawn using these empty background images as base maps during the generation of the second video. This allows the generation of the second video to focus on generating only the objects performing the actions, without generating backgrounds (such as complex backgrounds), thus improving the generation effect.

[0131] It should be noted that this application does not limit the method for identifying the foreground region; for example, it can employ any image segmentation processing method, such as... Figure 8 The image segmentation process shown can be implemented using any machine learning model with image segmentation capabilities. This image segmentation process can separate foreground and background regions from an image. Furthermore, this application does not limit the implementation method of this model.

[0132] In addition, to further improve the video generation effect, the second video mentioned above can be generated using a second model (such as...). Figure 7 The second video is generated by the model shown. It can be seen that, in one possible implementation, the generation process of the second video includes: inputting the at least one image, the empty background image of each image in the at least one image, and the skeleton image of each image in the first video into the second model, so that the encoding results of these empty background images, the encoding results of each image in the at least one image, and the encoding results of these skeleton images are concatenated at the channel level in the second model, and then generated by the second model except for the image encoder (such as...). Figure 7 Other networks besides the image encoder shown process the stitched result to obtain the second video (such as...). Figure 7 (See video 2). Because the second model generates the video by creating objects on these empty background images, it only needs to generate the object performing the action, without needing to generate a background (such as a complex background). This allows the second model to complete the video generation task even when it does not support generating complex backgrounds, thus reducing the difficulty of obtaining the second model and improving the universality of the video generation method provided in this application.

[0133] In addition, to further improve the video generation effect, this application also provides a possible implementation of the second model described above. In this implementation, the second model can be trained based on a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video, so that the second model can better learn from the sample video and its corresponding data how to generate a new video based on an image, an empty background image of the image, and a video.

[0134] It should be noted that this application does not limit the training process of the second model. For example, it may specifically include: acquiring a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video; then inputting the skeleton image of each image in the sample video, the reference image, and the empty background image into the second model to obtain the generated video output by the second model; updating the second model based on the difference between the generated video and the sample video so that the updated second model has better video generation performance, so that the updated second model can be used to continue to execute the aforementioned step of "acquiring a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video" and subsequent steps to achieve the next round of training process. This process is iterated and looped until the second stopping condition is met (such as the model loss of the second model being lower than the second loss threshold, the rate of change of the model loss being lower than the second rate of change threshold, the number of updates of the second model reaching the second number threshold, etc.), at which point the multi-round iterative training process for the second model ends. The model loss indicates the performance of the second model; and the model loss is determined based on the difference between the generated video and the sample video. Furthermore, this application does not limit the calculation process of the model loss.

[0135] Furthermore, this application does not limit the method of obtaining the aforementioned "sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video." For example, it may specifically include: after randomly selecting a video from a video library as a third video, extracting video segments (such as all or part of the third video) from the third video to obtain the sample video; extracting a first image from the third video to obtain the reference image corresponding to the sample video; and using the empty background image of the first image as the empty background image of the reference image corresponding to the sample video. It should be noted that this application does not limit the first image. For example, the first image may refer to an image randomly selected from the third video, or it may refer to an image selected from the third video that satisfies the restriction that "the pose of the object in the image is simple and the body of the object in the image is complete."

[0136] Research has found that, in the data acquisition process shown above, the empty background image of the reference image corresponding to the sample video is obtained by processing the reference image so that the empty background image not only carries the background information of the reference image, but may also carry the edge contour information of the object in the reference image. As a result, the second model learns some incorrect knowledge through the edge contour information, which in turn has some adverse effects on the training effect of the second model.

[0137] Based on the above research, in order to overcome the adverse effects shown in the preceding paragraph, this application also provides a method for obtaining the aforementioned "sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video". Specifically, this method may include: extracting a video segment and a first image from a third video, and extracting a second image from a fourth video, wherein the empty background image of the second image differs from the empty background image of the first image; performing background replacement processing on each image in the video segment based on the empty background image of the second image to obtain the sample video, so that the background of each image in the sample video is consistent with the background described by the empty background image of the second image; and performing background replacement processing on the first image based on the empty background image of the second image to obtain the reference image corresponding to the sample video, so that the background of the reference image is consistent with the background described by the empty background image of the second image. By maintaining a consistent background description and using the empty background image of the second image as the empty background image of the reference image corresponding to the sample video, information leakage caused by the removal of foreground region edge contours in the empty background image included in the original training data can be overcome by replacing the background in the obtained original training data with other backgrounds (such as replacing a background with a skirt outline with a trouser outline). This ensures that the second model can learn during training that there is no relationship between the edge contour information carried by the empty background image and the final generated video. This ensures that the trained second model can directly ignore the edge contour information carried by the empty background image when generating the video, thereby enabling the trained second model to better learn how to generate a new video based on an image, the empty background image of that image, and a video.

[0138] It should be noted that this application does not limit the timing of the background replacement process shown in the previous paragraph. For example, in some scenarios, the background replacement process needs to be executed in each round of training of the second model to improve its training performance. Furthermore, in some scenarios, to further improve training efficiency, the background replacement process can be executed with a certain probability (e.g., a probability of 0.2) in each round of training of the second model, while the data acquisition process shown in the third paragraph above is executed with the remaining probability (e.g., a probability of 0.8). This approach better balances the training performance and efficiency of the second model.

[0139] It should also be noted that the aforementioned fourth video refers to a video selected from a pre-built video library that is different from the third video, such that the fourth video differs from the third video in at least the following ways: the appearance (e.g., features, clothing, actions, etc.) of the objects in the fourth video differs from the appearance of the objects in the third video. Additionally, the second image refers to an image randomly selected from the fourth video.

[0140] It should also be noted that this application does not limit the above-described implementation of "there is a difference between the empty background image of the second image and the empty background image of the first image". For example, it may include: the edge contour of the foreground area indicated by the empty background image of the second image is different from the edge contour of the foreground area indicated by the empty background image of the first image.

[0141] It should also be noted that, in order to improve efficiency, when the background in the second image is the same as the background of each image in the third video, the method of obtaining the "sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video" may include: extracting a video segment from the third video to obtain the sample video, extracting a first image from the third video to obtain the reference image corresponding to the sample video, so that the edge contour of the object in the reference image is different from the edge contour of the object in the second image extracted from the fourth video, and using the empty background image of the second image as the empty background image of the reference image corresponding to the sample video. This is beneficial to improving the efficiency of training data acquisition.

[0142] In addition, to further improve the reproduction performance of the second model in some complex actions (such as turning actions), this application also provides a possible implementation of the sample video used in one round of training of the second model. In this method, the acquisition process of the sample video includes: extracting a video segment from a third video, wherein the object is in a frontal view in the first frame of the video segment and in a rear view in the last frame of the video segment; and determining the sample video based on the video segment so that the sample video can describe the complete turning action, thereby enabling the training data constructed based on the sample video to better describe the characteristics of the turning action, and thus enabling the second model to learn how to better generate turning actions from the training data, which is beneficial to improving the generation performance of the second model in turning scenes.

[0143] As can be seen, in one possible implementation, for the current round of training of the second model, after randomly selecting a video from the video library as the third video, the frontal and back images are first identified from the third video based on the skeleton diagrams of each image in the third video, so that the object in the frontal image is in a frontal view and the object in the back image is in a back view; then, the video segment between the frontal and back images is extracted from the third video as a sample video, so that the object in the first frame of the sample video is in a frontal view and the object in the last frame of the sample video is in a back view, so that the sample video can describe a complete turning action; then, an image is randomly extracted from the third video as the reference image corresponding to the sample video, and the empty background image of the randomly extracted image is used as the empty background image of the reference image corresponding to the sample video, so that the second model can be trained subsequently based on the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video.

[0144] It should be noted that the process of acquiring training data shown in the above two paragraphs can occur with a certain probability during the multiple rounds of iterative training of the second model. For example, the acquisition process can occur with a probability of 0.2.

[0145] Furthermore, in some scenarios, such as maintaining the background of a user-given video during video-driven processing, this application also provides a possible implementation of the aforementioned second video to better improve the video generation effect. In this implementation, the second video can also be generated based on the empty background map of each image in the first video, so that the background change indicated by the second video is consistent with the background change indicated by the empty background map of each image in the first video. This allows the second video to present the state of the object in the reference image under the background change indicated by the first video, thus better maintaining the background in the first video during video generation, thereby improving the video generation effect.

[0146] As can be seen, in one possible implementation, after obtaining the first video given by the user (e.g., ... Figure 2 Following the example video 1), firstly, skeleton diagrams and empty background images of each image in the first video are obtained, so that these skeleton diagrams can more accurately describe the action sequence indicated by the first video, and these empty background images can more accurately describe the background changes indicated by the first video; then, based on these skeleton diagrams, these empty background images, and at least one of the above-mentioned images (such as...) obtained by optimizing a reference image given by the user... Figure 2(Multiple images shown) are used to generate a second video, so that the generation process of the second video can be achieved by generating objects on these empty background images. This means that only objects need to be generated in the generation process of the second video, without the need to generate the background. This effectively overcomes the defects caused by the complex background in the first video, thereby improving the background preservation effect and improving the video generation effect.

[0147] It should be noted that this application does not limit the generation method of the second video shown in the above paragraph. For example, when the second video is generated using the second model, the generation process of the second video may include: inputting the skeleton diagram of each image in the first video, the empty background diagram of each image in the first video, and at least one of the above images into the second model, so that the second model follows the... Figure 7 The processing procedure shown yields and outputs the second video containing the second model.

[0148] It should also be noted that for any image (such as the images in the first video), the empty background image can describe the background characteristics of the image. This allows for image generation based on the empty background image to be achieved by adding objects to the empty background image, effectively reducing the amount of generated content. This not only improves generation efficiency but also enhances background preservation, thus contributing to a better overall generation result. Please refer to the above text for details regarding the empty background image; for brevity, it will not be repeated here.

[0149] Based on the aforementioned video generation method, the solution provided in this application has the following characteristics: the solution can generate multiple new images with unchanged appearance and simple poses (such as the multiple images mentioned above) based on complex and diverse reference images given by the user, so that these new images and the reference images exhibit high consistency in texture details; moreover, the solution drives the objects in the reference images to perform the corresponding actions in the video (such as the first video mentioned above) given by the user, thereby realizing video generation processing in real-world scenarios. This enables the solution to generate more realistic and natural image-driven results in real-world scenarios with complex reference images, and also makes the solution exhibit strong stability in difficult scenarios such as turning around.

[0150] Based on the video generation method provided in the embodiments of this application, the embodiments of this application also provide a video generation apparatus, which will be discussed below. Figure 9 Explanation and clarification will be provided. Among them, Figure 9 This is a schematic diagram of a video generation apparatus provided in an embodiment of this application. It should be noted that for technical details of the video generation apparatus provided in this embodiment, please refer to the relevant content of the video generation method described above.

[0151] like Figure 9As shown, the video generation apparatus 900 provided in this application embodiment includes:

[0152] The data acquisition unit 901 is used to acquire a reference image, a first video, and at least one skeleton diagram, wherein different skeleton diagrams indicate different information, including actions and / or viewpoints.

[0153] The first generation unit 902 is configured to generate at least one image based on the reference image and the at least one skeleton diagram, wherein the at least one image indicates the state of an object in the reference image under the information indicated by each of the skeleton diagrams;

[0154] The second generation unit 903 is configured to generate a second video based on the at least one image and the first video, wherein the second video indicates the state of an object in the reference image under the action sequence indicated by the first video.

[0155] In one possible implementation, the first generation unit 902 is specifically configured to: in response to detecting that the reference image does not meet preset constraints, generate the at least one image based on the reference image and the at least one skeleton diagram, wherein each of the at least one image meets the preset constraints, the preset constraints indicating the requirements of the objects in the images used in the video generation process in terms of body posture and / or body integrity, and the images used indicating the appearance of the objects in the video obtained through the video generation process.

[0156] In one possible implementation, the at least one image is generated using a first model, which includes a main network and a reference network. The input data of the reference network includes the encoding result of the reference image. The main network processes the encoding result of the at least one skeleton map based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, which generates discretized features corresponding to each skeleton map. The reference network includes a second network layer corresponding to the self-attention module, which generates discretized features corresponding to the reference image. For any skeleton map in the at least one skeleton map, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to that skeleton map, the discretized features corresponding to other skeleton maps in the at least one skeleton map, and the discretized features corresponding to the reference image, to obtain the processed features corresponding to that skeleton map.

[0157] In one possible implementation, the at least one image is generated using a first model; the training process of the first model includes: acquiring at least one label image, a skeleton map of each label image, and a reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same as the object in each label image; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model to obtain the predicted image corresponding to each label image output by the first model; and updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image.

[0158] In one possible implementation, the training process is triggered according to a first probability, and the first model learns to perform image generation processing under controlled conditions through the training process.

[0159] In one possible implementation, the training process is triggered according to a second probability, and the first model learns to generate the same image under controlled conditions through the training process; each of the labeled images is the same image, and the difference between the state of the object in the same image and the state of the object in the reference image corresponding to the at least one labeled image reaches a preset condition.

[0160] In one possible implementation, the object in the reference image corresponding to the at least one label image is in a frontal view, and the object in the same image is in a rear view.

[0161] In one possible implementation, the training process is triggered according to a third probability, and the first model learns to perform image generation processing under uncontrolled conditions through the training process; obtaining at least one labeled image, the skeleton map of each labeled image, and the reference image corresponding to the at least one labeled image includes: obtaining the at least one labeled image and the skeleton map of each labeled image, wherein the objects in different labeled images are different; inputting the skeleton map of the at least one labeled image and the reference image corresponding to the at least one labeled image into the first model includes: inputting the skeleton map of the at least one labeled image into the first model, wherein the input data of the reference network in the first model is 0.

[0162] In one possible implementation, the training process is triggered according to a fourth probability, and the first model learns to generate images describing different objects under different control conditions through the training process; obtaining at least one label image, the skeleton map of each label image, and the reference image corresponding to the at least one label image includes: obtaining the at least one label image, the skeleton map of each label image, and the reference image matched by each label image, wherein the objects in different label images are different, and for any label image, the object in the label image and the object in the reference image matched by the label image are the same object; inputting the skeleton map of the at least one label image and the reference image corresponding to the at least one label image into the first model includes: inputting the skeleton map of the at least one label image and the reference image matched by the at least one label image into the first model.

[0163] In one possible implementation, the input data of the reference network in the first model includes the encoding result of the reference image matched by the at least one label image. The main network in the first model processes the encoding result of the skeleton map of the at least one label image based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, the first network layer generates discretized features corresponding to the skeleton map of each label image, and the reference network includes a second network layer corresponding to the self-attention module, the second network layer generates discretized features corresponding to the reference image matched by each label image. For any label image among the at least one label image, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to the skeleton map of the label image, the discretized features corresponding to the skeleton maps of other label images besides the label image among the at least one label image, and the discretized features corresponding to the reference image matched by the label image, to obtain the processed features corresponding to the skeleton map of the label image.

[0164] In one possible implementation, the training process further includes: obtaining a first text corresponding to the at least one labeled image and a second text corresponding to the at least one labeled image, wherein the first text and the second text describe the at least one labeled image; the input data of the first model further includes the first text and the second text, wherein the input data of the reference network in the first model includes at least the encoding result of the second text, and the input data of the main network in the first model includes at least the encoding result of the first text.

[0165] In one possible implementation, the training process further includes: adding new modules to each attention module in the first model; updating the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image includes: updating each of the new modules in the first model based on the difference between the predicted image corresponding to the at least one label image and the at least one label image.

[0166] In one possible implementation, the second video is further generated based on the empty background image of each of the at least one image; for any of the at least one images, the process of obtaining the empty background image of the image includes: identifying a foreground region from the image, the foreground region indicating the position of an object in the image; removing the foreground region from the image; and performing image restoration processing on the image to obtain the empty background image of the image.

[0167] In one possible implementation, the second video is generated using a second model trained on a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video. The process of obtaining the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video includes: extracting a video segment from a third video to obtain the sample video; extracting a first image from the third video to obtain the reference image corresponding to the sample video; and using the empty background image of the first image as the empty background image of the reference image corresponding to the sample video; or, the sample video, the reference image, and the empty background image of the reference image corresponding to the sample video are all generated using a second model trained on a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video. The process of obtaining the reference image corresponding to the sample video and the empty background image of the reference image corresponding to the sample video includes: extracting the video segment and the first image from the third video, and extracting the second image from the fourth video, wherein there is a difference between the empty background image of the second image and the empty background image of the first image; performing background replacement processing on each image in the video segment based on the empty background image of the second image to obtain the sample video; performing background replacement processing on the first image based on the empty background image of the second image to obtain the reference image corresponding to the sample video; and using the empty background image of the second image as the empty background image of the reference image corresponding to the sample video.

[0168] In one possible implementation, the second video is generated using a second model trained on a sample video; the process of acquiring the sample video includes: extracting a video segment from a third video, wherein the object in the first frame of the video segment is in a frontal view and the object in the last frame of the video segment is in a rear view, and determining the sample video based on the video segment.

[0169] In one possible implementation, the at least one skeleton map includes a skeleton map of an image selected from the first video; and / or, the second video is generated based on the skeleton maps of each image in the first video.

[0170] Based on the aforementioned content regarding the video generation device 900, its working principle includes: first acquiring a reference image (such as an image provided by the user that indicates the appearance of an object in the final generated image), a first video (a video provided by the user indicating a sequence of actions), and at least one skeleton diagram (such as a skeleton diagram describing some simple actions), with different skeleton diagrams indicating different information (such as actions and / or perspectives); then generating at least one image (such as some images that can present simple actions and have a complete frame) based on the reference image and the at least one skeleton diagram, so that the at least one image can indicate the state of the object in the reference image under the information indicated by each skeleton diagram, thereby enabling the at least one image to better present the characteristics of the object; then generating a second video based on the at least one image and the first video, so that the second video can indicate the state of the object in the reference image under the action sequence indicated by the first video, thus enabling the object in the reference image to perform the action sequence indicated by the video. In this context, because the at least one skeleton diagram can indicate some simple actions, the at least one image generated based on the at least one skeleton diagram and the reference image can more accurately and comprehensively describe the characteristics of the object in the reference image, thereby enabling the video generated based on the at least one image to better preserve the characteristics of the object, which is beneficial to improving the video generation effect.

[0171] In addition, this application also provides an electronic device, the device including a processor and a memory: the memory is used to store instructions or computer programs; the processor is used to execute the instructions or computer programs in the memory, so that the electronic device performs any implementation of the video generation method provided in this application.

[0172] See Figure 10 The diagram illustrates a structural schematic of an electronic device 1000 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0173] like Figure 10As shown, the electronic device 1000 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device 1000. The processing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0174] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0175] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0176] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0177] This application also provides a computer-readable medium storing instructions or a computer program that, when executed on a device, causes the device to perform any implementation of the video generation method provided in this application.

[0178] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0179] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0180] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0181] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0182] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0184] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0185] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0186] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0187] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0188] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0189] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0190] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0191] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video generation method, characterized in that, The method includes: Acquire a reference image, a first video, and at least one skeleton diagram, wherein different skeleton diagrams indicate different information, including action and / or viewpoint; Based on the reference image and the at least one skeleton diagram, at least one image is generated, the at least one image indicating the state of an object in the reference image under the information indicated by each of the skeleton diagrams; Based on the at least one image and the first video, a second video is generated, the second video indicating the state of the object in the reference image under the action sequence indicated by the first video.

2. The method according to claim 1, characterized in that, The step of generating at least one image based on the reference image and the at least one skeleton map includes: In response to the detection that the reference image does not meet the preset constraints, at least one image is generated based on the reference image and the at least one skeleton diagram, wherein each of the at least one image meets the preset constraints, the preset constraints indicating the requirements of the object in the image used in the video generation process in terms of body posture and / or body integrity, and the image used indicating the appearance of the object in the video obtained through the video generation process.

3. The method according to claim 1, characterized in that, The at least one image is generated using a first model, which includes a main network and a reference network. The input data of the reference network includes the encoding result of the reference image, and the main network processes the encoding result of the at least one skeleton map based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, the first network layer generates discretized features corresponding to each skeleton map, and the reference network includes a second network layer corresponding to the self-attention module, the second network layer generates discretized features corresponding to the reference image. For any skeleton in the at least one skeleton, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to the skeleton, the discretized features corresponding to other skeletons in the at least one skeleton besides the skeleton, and the discretized features corresponding to the reference image, so as to obtain the processed features corresponding to the skeleton.

4. The method according to claim 1, characterized in that, The at least one image is generated using the first model; The training process of the first model includes: Obtain at least one label image, a skeleton diagram of each label image, and a reference image corresponding to the at least one label image, wherein the object in the reference image corresponding to the at least one label image is the same object as the object in each label image; The skeleton diagram of the at least one label image and the reference image corresponding to the at least one label image are input into the first model to obtain the predicted image corresponding to each label image output by the first model. The first model is updated based on the difference between the predicted image corresponding to the at least one labeled image and the at least one labeled image.

5. The method according to claim 4, characterized in that, The training process is triggered according to a first probability, and the first model learns to perform image generation processing under controlled conditions through the training process.

6. The method according to claim 4, characterized in that, The training process is triggered according to a second probability, and the first model learns to generate the same image under controlled conditions through the training process. All the labeled images are the same image, and the difference between the state of the object in the same image and the state of the object in the reference image corresponding to the at least one labeled image reaches a preset condition.

7. The method according to claim 6, characterized in that, The object in the reference image corresponding to the at least one label image is in a frontal view, and the object in the same image is in a rear view.

8. The method according to claim 4, characterized in that, The training process is triggered according to a third probability, and the first model learns to perform image generation processing under uncontrolled conditions through the training process. The step of obtaining at least one label image, a skeleton diagram of each label image, and a reference image corresponding to the at least one label image includes: Obtain the at least one label image and the skeleton diagram of each label image, wherein the objects in the different label images are different; The step of inputting the skeleton diagram of the at least one label image and the reference image corresponding to the at least one label image into the first model includes: The skeleton diagram of the at least one labeled image is input into the first model, where the input data of the reference network in the first model is 0.

9. The method according to claim 4, characterized in that, The training process is triggered according to the fourth probability, and the first model learns to generate images describing different objects under different control conditions through the training process. The step of obtaining at least one label image, a skeleton diagram of each label image, and a reference image corresponding to the at least one label image includes: Obtain at least one label image, a skeleton diagram of each label image, and a reference image matching each label image. The objects in different label images are different. For any label image, the object in the label image and the object in the reference image matching the label image are the same object. The step of inputting the skeleton diagram of the at least one label image and the reference image corresponding to the at least one label image into the first model includes: The skeleton diagram of the at least one label image and the reference image matching the at least one label image are input into the first model.

10. The method according to claim 9, characterized in that, The input data of the reference network in the first model includes the encoding result of the reference image matched by the at least one label image. The main network in the first model processes the encoding result of the skeleton map of the at least one label image based on the output data of the reference network. For any self-attention module in the main network, the main network includes a first network layer corresponding to the self-attention module, the first network layer generates discretized features corresponding to the skeleton map of each label image, and the reference network includes a second network layer corresponding to the self-attention module, the second network layer generates discretized features corresponding to the reference image matched by each label image; For any of the at least one label images, the self-attention module performs self-attention processing on the concatenation result between the discretized features corresponding to the skeleton map of the label image, the discretized features corresponding to the skeleton maps of the other label images in the at least one label image besides the label image, and the discretized features corresponding to the reference image matched by the label image, so as to obtain the processed features corresponding to the skeleton map of the label image.

11. The method according to claim 4, characterized in that, The training process also includes: Obtain a first text corresponding to the at least one label image and a second text corresponding to the at least one label image, wherein the first text and the second text describe the at least one label image; The input data of the first model also includes the first text and the second text. The input data of the reference network in the first model includes at least the encoding result of the second text, and the input data of the main network in the first model includes at least the encoding result of the first text.

12. The method according to claim 11, characterized in that, The training process also includes: Add new modules to the attention modules of the first model respectively; The step of updating the first model based on the difference between the predicted image corresponding to the at least one labeled image and the at least one labeled image includes: Based on the difference between the predicted image corresponding to the at least one labeled image and the at least one labeled image, each of the new modules in the first model is updated.

13. The method according to claim 1, characterized in that, The second video is also generated based on the empty background image of each of the at least one image; For any one of the at least one images, the process of obtaining the empty background image of that image includes: Identify the foreground region from the image, the foreground region indicating the location of an object in the image; Remove the foreground region from the image; The image is then inpainted to obtain an empty background image.

14. The method according to claim 13, characterized in that, The second video is generated using a second model, which is trained based on a sample video, a reference image corresponding to the sample video, and an empty background image of the reference image corresponding to the sample video. The process of obtaining the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video includes: The sample video is obtained by extracting a video segment from the third video, the first image is obtained from the third video to obtain a reference image corresponding to the sample video, and the empty background image of the first image is used as the empty background image of the reference image corresponding to the sample video. or, The process of obtaining the sample video, the reference image corresponding to the sample video, and the empty background image of the reference image corresponding to the sample video includes: Extract the video segment and the first image from the third video, and extract the second image from the fourth video, wherein there is a difference between the empty background image of the second image and the empty background image of the first image; Based on the empty background image of the second image, background replacement processing is performed on each image in the video segment to obtain the sample video. Based on the empty background image of the second image, background replacement processing is performed on the first image to obtain the reference image corresponding to the sample video. The empty background image of the second image is used as the empty background image of the reference image corresponding to the sample video.

15. The method according to claim 13, characterized in that, The second video was generated using a second model, which was trained based on sample videos; The process of acquiring the sample video includes: extracting a video segment from a third video, wherein the object in the first frame of the video segment is in a frontal view and the object in the last frame of the video segment is in a rear view, and determining the sample video based on the video segment.

16. The method according to any one of claims 1-15, characterized in that, The at least one skeleton diagram includes a skeleton diagram of an image selected from the first video; And / or, The second video is generated based on the skeleton diagrams of each image in the first video.

17. A video generation apparatus, characterized in that, include: A data acquisition unit is used to acquire a reference image, a first video, and at least one skeleton diagram, wherein different skeleton diagrams indicate different information, including actions and / or viewpoints. The first generation unit is configured to generate at least one image based on the reference image and the at least one skeleton diagram, wherein the at least one image indicates the state of an object in the reference image under the information indicated by each of the skeleton diagrams; The second generation unit is configured to generate a second video based on the at least one image and the first video, wherein the second video indicates the state of an object in the reference image under the action sequence indicated by the first video.

18. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-16.

19. A computer-readable medium, characterized in that, The computer-readable medium stores instructions or computer programs that, when executed on the device, cause the device to perform the method according to any one of claims 1-16.

20. A computer program product, characterized in that, It includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the method of any one of claims 1-16.

Images