Image processing method and apparatus, device, medium, and program product

Abstract

Disclosed in embodiments of the present application are an image processing method and apparatus, a device, a medium, and a program product. The method comprises: acquiring a reference image and a content description text; performing encoding and copying processing on the reference image to obtain N image encoded features to be attenuated; performing feature attenuation processing on said N image encoded features to obtain N attenuated image encoded features; and according to image content described in the content description text, performing video generation processing on the N attenuated image encoded features to obtain a target video. By applying the embodiments of the present application, it can be ensured that image content of each video frame in the target video and image content of the reference image have both content diversity and content consistency.

Description

An image processing method, apparatus, device, medium, and program product

[0001] This application claims priority to Chinese Patent Application No. 2024112071482, filed on August 30, 2024, entitled “An Image Processing Method, Apparatus, Device, Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and more particularly to the field of artificial intelligence, specifically to an image processing method, apparatus, device, medium, and program product. Background Technology

[0003] Image-to-video conversion refers to the process of generating a video that matches the content of a single image.

[0004] Currently, diffusion models are supported for generating videos based on images. However, adding a time-series dimension to a diffusion model can lead to excessive loss of content information in the reference image, causing inconsistencies between the image content of each frame in the video and that of the reference image. Similarly, adding only the encoded features of the reference image to the latent feature generation stage can result in inconsistencies between the noise reduction generation stage and the reference image content. Furthermore, both of these related techniques carry the risk of producing video frames with identical image content in the generated frame sequence, resulting in a lack of content variation in the generated video.

[0005] Therefore, how to convert images into videos has become a research hotspot. Summary of the Invention

[0006] This application provides an image processing method, apparatus, device, medium, and program product that can ensure that the image content of each video frame in the target video matches the video content described in the content description text, while ensuring that the image content of each video frame has both content differences and content consistency with the image content of the reference image.

[0007] On one hand, embodiments of this application provide an image processing method, the method comprising:

[0008] Obtain a reference image and content description text. The reference image is used to generate the target video, which consists of a frame sequence of N video frames. The content description text is used to describe the video content that the target video should contain. N is a positive integer.

[0009] The reference image is encoded and copied to obtain N encoding features of the image to be attenuated. The encoding features of the image to be attenuated are used to characterize the content characteristics of the reference image content.

[0010] Feature attenuation processing is performed on N image coding features to be attenuated to obtain N attenuated image coding features; one attenuated image coding feature is used to generate one video frame; the feature attenuation processing is used to control the content difference between the image content of each video frame and the reference image content;

[0011] Based on the content description text, video generation processing is performed on N attenuated image coding features to generate the target video; the image content of each frame in the generated target video matches the video content, and there is a content difference between the target video and the reference image content.

[0012] On the other hand, embodiments of this application provide an image processing apparatus, which includes:

[0013] The acquisition unit is used to acquire a reference image and content description text. The reference image is used to generate the target video, which includes a frame sequence consisting of N video frames. The content description text is used to describe the video content that the target video should contain. N is a positive integer.

[0014] The processing unit is used to perform encoding and copying processing on the reference image to obtain N image encoding features to be attenuated. The image encoding features to be attenuated are used to characterize the content characteristics of the reference image content.

[0015] The processing unit is also used to perform feature attenuation processing on N image coding features to be attenuated, to obtain N attenuated image coding features; one attenuated image coding feature is used to generate a video frame; the feature attenuation processing is used to control the content difference between the image content of each video frame and the reference image content;

[0016] The processing unit is also used to perform video generation processing on N attenuated image coding features according to the content description text to generate a target video; the image content of each video frame in the generated target video matches the video content and has content differences from the reference image content.

[0017] On the other hand, embodiments of this application provide a computer device, which includes:

[0018] A processor, adapted to execute computer programs;

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image processing method.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described image processing method.

[0021] On the other hand, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image processing method.

[0022] In this embodiment, a reference image is obtained as guidance information for generating the target video, and content description text is obtained to describe the video content that the target video should contain. Then, the reference image is subjected to copy encoding processing to obtain N image encoding features to be attenuated; each image encoding feature to be attenuated is used to characterize the content characteristics of the reference image content, and there is a one-to-one correspondence between the N image encoding features to be attenuated and the number N of video frames in the frame sequence to be generated. Feature attenuation processing is performed on the N image encoding features to be attenuated to obtain N attenuated image encoding features. On the one hand, each attenuated image encoding feature and the image encoding features to be attenuated have feature differences, thereby ensuring that the N video frames generated based on the N attenuated image encoding features with feature differences have content differences. Simultaneously, relying on the N attenuated image encoding features to characterize the content characteristics of the reference image content to different degrees, the image content of each video frame in the target video generated based on the N attenuated image encoding features has content consistency with the image content of the reference image. On the other hand, during the generation of N video frames from the reference image, the reference to the reference image is reduced frame by frame. This allows for the introduction of more image content described in the content description text to constrain / guide the generation of the target video. This not only ensures that the image content of each video frame in the target video frame sequence matches the image content described in the content description text (e.g., the appearance of the subject in the video frame matches the action described in the content description text), but also that the video content described in the content description text can assist in the generation of image elements in the content differences between the video frame's image content and the reference image's image content. Furthermore, each of the N attenuated image coding features is used to generate a video frame. Since the N attenuated image coding features are not the same, this ensures that the image content of the N video frames is also different, thereby ensuring that the target video can present content variation effects. Attached Figure Description

[0023] Figure 1a is a schematic diagram of a training framework for a Stable Diffusion model provided in an exemplary embodiment of this application;

[0024] Figure 1b is a schematic diagram of the reasoning framework of a Stable Diffusion model provided by an exemplary embodiment of this application;

[0025] Figure 2 is a schematic diagram of the model structure of a video generation model obtained by embedding a spatiotemporal attention module in a Stable Diffusion model, as provided in an exemplary embodiment of this application.

[0026] Figure 3 is a schematic diagram of the architecture of an image processing system in the case of a latent space embedding mechanism provided by an exemplary embodiment of this application.

[0027] Figure 4a is a schematic diagram of a model structure after fine-tuning the feature extraction module in a video generation model, provided by an exemplary embodiment of this application;

[0028] Figure 4b is a schematic diagram of the architecture of an image processing system provided by an exemplary embodiment of this application, in the case of a multi-level cross-attention mechanism for feature embedding.

[0029] Figure 5 is a schematic diagram of the architecture of an image processing system under a dual-stream consistent embedding mechanism, provided by an exemplary embodiment of this application.

[0030] Figure 6 is a schematic diagram of the scenario architecture of an application scenario provided by an exemplary embodiment of this application;

[0031] Figure 7 is a flowchart illustrating an image processing method provided in an exemplary embodiment of this application;

[0032] Figure 8 is a schematic diagram of an interface for uploading a reference image and content description text provided in an exemplary embodiment of this application;

[0033] Figure 9 is a schematic diagram of the calculation flow of feature attenuation processing provided in an exemplary embodiment of this application;

[0034] Figure 10 is a schematic diagram of embedding N attenuated latent space reference features into N latent space noise initialization features according to an exemplary embodiment of this application;

[0035] Figure 11 is a schematic flowchart of another image processing method provided in an exemplary embodiment of this application;

[0036] Figure 12 is a schematic diagram of the process of image encoding processing of a reference image by an image encoder used in a multi-level cross-attention embedding mechanism according to an exemplary embodiment of this application.

[0037] Figure 13 is a schematic diagram of another image encoding process for a reference image under a multi-level cross-attention embedding mechanism provided by an exemplary embodiment of this application;

[0038] Figure 14 is a schematic diagram of the structure of a fine-tuned feature extraction module provided in an exemplary embodiment of this application;

[0039] Figure 15 is a flowchart illustrating another image processing method provided in an exemplary embodiment of this application;

[0040] Figure 16 is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this application;

[0041] Figure 17 is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0042] This application proposes an image processing scheme, specifically an image-to-video generation scheme. Image-to-video refers to the process of converting a static image into a dynamic video; that is, providing a static image as a content reference, and using this static image as a reference to generate a corresponding video. The resulting video comprises a frame sequence of N video frames, where N is a positive integer. Thus, the process of converting the image essentially generates N video frames based on the image, and these N video frames are arranged in chronological order to form the video (i.e., a video frame sequence, or simply a frame sequence).

[0043] In image-to-video conversion technology, it is necessary to ensure both consistency and difference in the image content of each video frame in the frame sequence and the image content of the static image. Specifically, ① the image content of each video frame in the resulting frame sequence must maintain consistency with the image content of the static image. Content consistency can be understood as the main subject (the main content) in each video frame being the same as the main subject in the static image. The main subject in the image content (the main object in the image or video frame) can be understood as the primary object in the image or video frame, i.e., the object that the image or video frame wants to highlight. For example, in a person image, the object to be highlighted is the person; therefore, after generating a video based on the person image, it is necessary to ensure that the person included in each frame of the video is the same as the person in the person image, such as having the same appearance, clothing, etc., and that it is possible to identify that the person in the video frame and the person in the person image are the same person. In a landscape image, the objects to be highlighted are mountains and lakes, etc. Therefore, after generating a video based on the landscape image, it is necessary to ensure that the mountains and lakes included in each frame of the video are the same as those in the landscape image, such as the relative positions between the mountains and lakes are the same, the shapes of the mountains are the same, the shapes of the lakes are the same, and so on.

[0044] ② The image content of each video frame in the frame sequence obtained from image-to-video conversion needs to maintain a content difference from the image content of the static image. Content difference can be understood as the difference in the way the subject is presented in each video frame differs from the way the subject is presented in the static image. For example, when the static image is a person, the person's expression can include, but is not limited to, facial expressions (such as smiling, pouting, blinking, and pursing their lips) and limb movements (such as walking, standing still, jumping, and running). Similarly, when the static image is a landscape image, and the main subjects are mountains and lakes, the expression of the mountains can include objects on the mountains (such as trees and grass) swaying in the wind, and the expression of the lake can include the movement of water ripples on the lake surface. By controlling the content difference between the static image content and the image content of each video frame, it is ensured that the video presents a dynamic visual effect, thus meeting the requirements for image-to-video conversion based on static images to generate dynamic video.

[0045] To ensure that the video generated by image-to-video technology meets the requirements of content consistency and content differentiation; that is, to ensure that the video content generated by image-to-video technology is both based on the desired main content and can present the desired dynamic effects, the image-to-video generation scheme provided in this application embodiment is specifically an image-to-video generation scheme based on attenuation feature injection. The general principle of this scheme can include:

[0046] A reference image and content description text are obtained. The reference image is used to generate the target video (the target video consists of a frame sequence of N video frames). Specifically, the main content of the reference image is used to generate the main content of each video frame in the frame sequence, ensuring content consistency between the image content of each video frame and the reference image content. The content description text describes the video content that the target video to be generated should contain. Specifically, it describes the image content that each video frame in the corresponding frame sequence should contain, guiding the trend of the subject's appearance changes in the image content of each video frame, such as changes in the movement of people in the video frame. Then, the reference image is encoded and copied to obtain N image encoding features to be attenuated. These features characterize the content characteristics of the reference image. In other words, the encoding and copying process of the reference image aims to extract the features of the main subject in the image content of the reference image and copy these features into N copies, each corresponding to one video frame. Next, feature attenuation processing is applied to the N image coding features to be attenuated, resulting in N attenuated image coding features. This feature attenuation processing can be used to control the content difference between the image content of each video frame in the frame sequence and the reference image content, that is, to ensure that when the attenuated image coding is used to generate video frames, the image content of the video frame can maintain a certain content difference with the image content of the reference image. In this way, after processing the video generation using the N attenuated image coding features according to the video content described in the content description text, a target video can be obtained in which the image content of each video frame has both content difference and content consistency with the reference image content. The role of the content description text in the video generation process is to introduce the video content described in the content description text when generating video frames with content differences from the reference image, thereby guiding the points or trends of content difference changes; for example, the points of difference between the facial features (such as the curvature of the lips) in the image content of the reference image and the facial features in the image content of the video frame are guided by the content description text.

[0047] Therefore, on the one hand, this application embodiment introduces a reference image (referred to as ref image) as the base image for generating the target video; the base image serves as reference information throughout the entire video generation process, that is, the content information in this base image is referenced throughout the entire frame sequence generation process, ensuring that the image content of each video frame in the generated frame sequence has content consistency with the image content in the base image, preventing sudden changes in the video content of the target video, and greatly improving the content retention of the overall content of the target video to the image content of the reference image. On the other hand, the image encoding features to be attenuated extracted from the reference image need to be attenuated before being embedded into the video generation process. This attenuation-based embedding method can attenuate the guidance information (i.e., the image encoding features to be attenuated) calculated based on the ref image over time (such as the playback time of the frame sequence). This allows the generation of image content in video frames corresponding to different timestamps to be guided by the attenuated image encoding features, ensuring that there is a content difference between the image content of each video frame in the frame sequence and the image content of the reference image. Furthermore, the differences in content between adjacent video frames are guided by the video content described in the content description text, thereby improving the video quality of the target video, specifically ensuring that the overall video content of the target video meets the user's requirements.

[0048] The image processing scheme proposed in this application is based on a video generation model. Specifically, after obtaining N attenuated image coding features of the reference image, these N attenuated image coding features need to be embedded into the process of calling the video generation model for video generation processing to convert the reference image into a target video. The video generation model is an improvement on the basic diffusion model; the improvements are: adding a spatiotemporal attention module to the basic diffusion model, and adding an attention mechanism to the basic diffusion model. To better understand the architecture and principles of the video generation model, the basic diffusion model and the spatiotemporal attention module are introduced below:

[0049] (1) Diffusion model.

[0050] A diffusion model (or simply text-based image diffusion model) is a pre-trained model that can be viewed as a series of denoising autoencoders. Its purpose is to eliminate the continuous application of Gaussian noise to the training images. During the pre-training process, the input text controls the generation of images. Specifically, the diffusion model includes two main processes: forward diffusion and backward diffusion. In the forward diffusion stage, the image is gradually contaminated with noise until it becomes completely random noise. In the backward diffusion process, a series of Markov chains are used to progressively remove the predicted noise at each time step (or timestamp, such as 1 second), thereby recovering the data from the Gaussian noise to generate the image. Furthermore, there are many types of diffusion models. This application uses a Latent Diffusion Model (LDM) as an example, which can be called a Stable Diffusion model. It diffuses in the latent space rather than the pixel space, saving memory. Combined with textual semantic feedback from the Transformer, it can produce diverse and highly detailed images while preserving the semantic structure of the data.

[0051] The basic training framework of the Stable Diffusion model can be seen in Figure 1a, and the basic inference framework can be seen in Figure 1b. As shown in Figures 1a and 1b, both the basic training and inference frameworks of the Stable Diffusion model include the following three parts: a variational autoencoder, a U-Net network, and a text encoder. Specifically:

[0052] ① A variational autoencoder consists of two parts: an encoder and a decoder. The encoder part of a variational autoencoder can be called a VAE encoder, and the decoder part can be called a VAE decoder.

[0053] The encoder (vae) transforms a reference image into a latent representation in a low-dimensional latent space (similar to the use of variational autoencoders in traditional Stable Diffusion models: the encoder is used only during the training phase to encode sample images, not during inference). This latent representation serves as the input data for the U-Net network. The decoder (vae) operates in the opposite way, transforming the latent representation back into a high-dimensional image (i.e., video frames). The latent representation is a low-dimensional vector learned from the reference image, representing its inherent and implicit features. In Stable Diffusion models, latent representations typically refer to latent variables—variables not directly observed in the model. These can be understood as the most fundamental features throughout the entire model processing. In Stable Diffusion models, latent variables may represent unobserved factors such as individual characteristics, behavior, style, and composition, which can influence the diffusion process. The information space in which the latent representation resides is called the latent space. The latent space, also known as the latent space mentioned above, is a low-dimensional vector space. The latent representation in this space represents a highly generalized or compressed form of the image features in the reference image. This generalization or compression is achieved by the encoder VAE encoder, which projects the reference image into the latent space to obtain the latent representation of the image features of the reference image, thereby mining more comprehensive, deeper, and implicit features in the reference image.

[0054] ②U-Net network.

[0055] The U-Net network, also known as the Unet network, is a semantic segmentation algorithm using a fully convolutional network. This algorithm incorporates a symmetrical U-shaped structure with compression and expansion paths. The Unet network includes an encoder (represented as downsampling) and a decoder (represented as upsampling). The encoder compresses the reference image into a low-resolution image, and the decoder decodes the low-resolution image back into a high-resolution image. The Unet network also includes skip connections, which are structures located between the encoder and decoder. These skip connections fuse features at different scales, thereby improving segmentation accuracy. Therefore, the Unet network consists of three parts: a downsampling part, a skip connection part, and an upsampling part. Each of these three parts has the same composition, differing only in the orientation of the feature size dimensions. Each part includes multi-level feature extraction modules (represented as Unet blocks), each consisting of a deep residual block (represented as ResNet) and an attention block. Deep residual blocks can introduce residual connections to solve problems such as gradient explosion and gradient vanishing during model training; attention blocks include cross attention, which is used to adjust the output of the content description text input to the model; attention blocks also include self attention, which is used to make the quality of video frames generated by the Unet network more refined, and self attention is located between deep residual blocks and cross attention.

[0056] The skip connection component is used to prevent the Unet network from losing important information during encoder downsampling, and this skip connection is typically added between the encoder's downsampling ResNet and the decoder's upsampling ResNet. Furthermore, cross-attention mechanisms are commonly used to handle the correlations between multiple input sequences, capturing important information interactions between different sequences in the model. In the Stable Diffusion model, cross-attention helps the model better understand the correlations between different parts, thus more accurately predicting changes and effects during the diffusion process. Self-attention is an important supplement to the Unet block in Stable Diffusion. Its input is the backbone features (i.e., latent representations) of the Stable Diffusion model. The role of self-attention is to further fuse all semantic representations in the backbone features, aiming to make the generated content more clearly represented and improve the Unet network's noise reduction capability for backbone features.

[0057] As shown in Figures 1a and 1b, during the training process of the Stable Diffusion model, the reference image is encoded with noise by the VAE encoder to generate a new latent representation. This latent representation is then input into the Unet network for noise reduction and other operations. The latent signal output by the Unet network is input into the VAE decoder for decoding to generate a frame sequence. During the inference process of the Stable Diffusion model, since there are no training images, its purpose is to generate specific image content based on the desired video content. Therefore, the input data of the Stable Diffusion model is the latent representation of pure noise, without the need for VAE encoder encoding. By inputting the latent representation of pure noise into the Unet network for noise reduction and other processing, and then having the VAE decoder decode the latent signal output by the Unet network, an image of the desired video content (in the basic diffusion model, this desired video content is described by text) can be generated.

[0058] ③Text encoder.

[0059] A text encoder transforms an input sequence of conceptual text into an embedding space that the UNet network can understand. This embedding vector space refers to text recognition vectors (represented as text embeddings). In this embodiment, the text encoder module of the clip model is supported as the text encoder. The text encoder module is a relatively simple encoder based on transformer (a deep learning model based on self-attention mechanism). In order to compute a good semantic representation of conceptual text, it maps the labeled sequence to a latent text embedding sequence, thereby obtaining a better expected output. The clip model is an advanced deep learning model capable of understanding text and images. Specifically, the clip model is a multimodal model. By training this multimodal model, it can match text and images, learn to recognize the content in the image and the language describing the image. During training, the clip model learns to encode images and text into a unified vector space, enabling it to understand the relationship between images and text in both linguistic and visual terms. Specifically, it learns to match images and text through contrastive learning, aiming to predict which image belongs to a given text, or which text belongs to a given image, by training the clip model. Therefore, the clip model can be used to identify elements such as objects, scenes, and actions in an image, and it can also understand text related to the image, such as labels, descriptions, and titles.

[0060] (2) Spatiotemporal attention module.

[0061] The spatiotemporal attention module, also known as the motion module, is used to enhance the video generation capabilities provided by the Stable Diffusion model. Its main function is to perform correlation calculations between frames in the frame sequence during the video generation and training process of the Stable Diffusion model. That is, to calculate the correlation between the image content of two adjacent video frames, thereby smoothing out the content gaps between generated frames.

[0062] Based on the above introduction to the basic diffusion model Stable Diffusion and the spatiotemporal attention module motion module, this application embodiment supports embedding the spatiotemporal attention module into the Stable Diffusion model. Specifically, the spatiotemporal attention module is embedded in the multi-level feature extraction modules of the Unet network in the Stable Diffusion model. A schematic diagram of the video generation model structure obtained by embedding the spatiotemporal attention module into the Stable Diffusion model can be seen in Figure 2. As shown in Figure 2, the method of adding the motion module to the Stable Diffusion model includes: embedding one motion module in each level of the multi-level feature extraction modules of the Stable Diffusion model, that is, embedding one motion module after each Unet block in the Stable Diffusion model to form a video generation model. In this way, during the generation of a frame sequence at once, after the Unet block module in the feature extraction module performs noise reduction and other processing on the latent representation, the spatiotemporal attention module embedded in the feature extraction module can be used to predict the output information (i.e., the latent representation) of the corresponding feature extraction module to obtain the prediction result. This prediction result is used to indicate the content correlation between the image content of two adjacent video frames when generating a frame sequence based on the output information of the corresponding feature extraction module. If the content correlation is too high, it indicates that the content jumps between frames are large and not smooth enough. In this case, the content changes can be appropriately slowed down during subsequent noise reduction and other processing to ensure the smoothness of the content between frames. It can be seen that by embedding the spatiotemporal attention module in the feature extraction module, the content correlation between video frames can be effectively calculated, so as to optimize the video frames based on the content correlation and avoid problems such as poor video effect caused by excessive content jumps between video frames. It should be noted that the functions of each module in the model structure of the video generation model shown in Figure 2 can be referred to the relevant descriptions in Figures 1a and 1b above, and will not be repeated here.

[0063] Furthermore, considering that the embodiments of this application introduce reference images for guiding content generation during the model inference stage, in order to embed the features of the reference images into the model inference process, the embodiments of this application will also fine-tune the video generation model shown in Figure 2 according to different feature embedding mechanisms for the reference images, so as to adaptively embed the features of the reference images into the model inference stage. Moreover, depending on the different feature embedding mechanisms for the reference images, the system architecture for the image-to-video generation scheme based on the video generation model also differs. The following describes the feature embedding mechanism for the reference images, the fine-tuning of the video generation model to adapt to this feature embedding mechanism, and the corresponding system architecture, wherein:

[0064] (1) The feature embedding mechanism is the latent space embedding mechanism.

[0065] The principle of the latent latent embedding mechanism is to embed the features of the reference image into N latent noise initialization features used when generating the frame sequence; each of these N latent noise initialization features is obtained by adding random noise to the blank latent reference features, which can be understood as the process of adding random noise to a blank image.

[0066] Since the features of the reference image are embedded into the latent space noise initialization features used when generating the sequence, no internal structural changes are required in the video generation model. Therefore, when the feature embedding mechanism is the latent space embedding mechanism, the corresponding video generation model structure is the same as that shown in Figure 2, avoiding the cost overhead of adjusting the model structure.

[0067] When the feature embedding mechanism is a latent space embedding mechanism, the system architecture of the image processing system can be seen in Figure 3. As shown in Figure 3, the image processing system includes at least a video generation model, input data for the video generation model (i.e., N latent space noise initialization features used when generating the frame sequence), and a first image processing module 301. This first image processing module 301 includes a VAE encoder and a feature attenuation submodule, which are built into the Stable Diffusion model. The VAE encoder performs image encoding on the reference image to map it to the latent space, obtaining N latent space reference features to be attenuated in the latent space to represent image features. The feature attenuation submodule performs feature attenuation processing on the N latent space reference features to be attenuated, obtaining N attenuated latent space reference features. These N attenuated latent space reference features are then embedded into the N latent space noise initialization features to obtain N latent space target features. After inputting N latent space target features into the video generation model, the video generation model can perform noise reduction and other processing based on the N latent space target features and content description text, and then use the VAE decoder to decode them, so as to generate the target video in one go based on the reference image and content description text.

[0068] Therefore, this latent embedding mechanism can directly pass the content information or features of the reference image to each latent space noise initialization feature used when generating the frame sequence. This is similar to directly passing the content information / features of the reference image to the N video frames during image generation, effectively ensuring content consistency between the image content of each video frame in the final generated target video and the image content of the reference image. Furthermore, a feature attenuation submodule is deployed in the first image processing module. This submodule can perform feature attenuation processing on the N latent space reference features extracted from the reference image, ensuring content differences between the image content of each video frame and the image content of the reference image. Further, the specific differences in content between the video frames are described by the content description text input to the video generation model, ensuring that the overall video content of the generated target video meets the video generation requirements and improving the video quality of the target video.

[0069] (2) The feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0070] The principle of the multi-level cross-attention embedding mechanism is to embed the features of the reference image into the feature extraction module of each level of the Unet network in the video generation model, specifically into the cross-attention mechanism of the feature frame extraction module of each level.

[0071] Since the features of the reference image are embedded into the video generation model, the module structure of the feature extraction module in the video generation model needs to be fine-tuned so that the fine-tuned feature extraction module can receive and process the features of the reference image. For example, a schematic diagram of the model structure of the video generation model after fine-tuning the feature extraction module can be seen in Figure 4a. As shown in Figure 4a, a cross-attention layer is added to each level of the feature extraction module in the Unet network. This ensures that the original cross-attention layer in the feature extraction module is used for attention operations on the content description text, while the newly added cross-attention layer is used for attention operations on the features of the reference image embedded in the feature extraction module.

[0072] When the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the system architecture of the image processing system can be seen in Figure 4b. As shown in Figure 4b, the image processing system includes at least a video generation model, input data for the video generation model (i.e., N latent space noise initialization features used when generating the frame sequence), and a second image processing module 302. The second image processing module 302 includes a novel image encoder (denoted as image encoder) and a feature attenuation submodule provided in this application embodiment. The image encoder encodes the reference image into specified N image feature vector embeddings to be attenuated, and then uses the cross-attention calculation in each level of the feature extraction module in the feature Unet network to combine the content information of the reference image during the video generation process.

[0073] Therefore, this multi-level cross-attention embedding mechanism can effectively ensure content consistency between the image content of each frame in the final generated target video and the image content of the reference image by passing the content information or features of the reference image to the video generation model during the video generation process. Furthermore, a feature attenuation submodule is deployed in the first image processing module. This submodule can perform feature attenuation processing on the N feature vectors of the image to be attenuated extracted from the reference image, thereby ensuring content differences between the image content of each video frame and the image content of the reference image. Further, the specific differences in content between the video frames are described by the content description text input to the video generation model, ensuring that the overall video content of the generated target video meets the video generation requirements and improving the video quality of the target video.

[0074] (3) The feature embedding mechanism is a two-stream consistent embedding mechanism (or a two-stream embedding mechanism), which includes both latent space embedding mechanism and multi-level cross-attention embedding mechanism.

[0075] The dual-stream consistency embedding mechanism refers to the process of embedding features from a reference image into the frame sequence generation process by simultaneously employing both latent space embedding and multi-level cross-attention embedding mechanisms during the model inference stage. Details regarding latent space embedding and multi-level cross-attention embedding mechanisms can be found in the aforementioned descriptions and will not be repeated here.

[0076] Considering that the dual-stream consistency embedding mechanism includes a multi-level cross-attention embedding mechanism, it is necessary to fine-tune the structure of the video generation model shown in Figure 2. The structural diagram of the fine-tuned video generation model is shown in Figure 4a above. The model structure of the fine-tuned video generation model will not be described in detail here.

[0077] When the feature embedding mechanism is a two-stream consistent embedding mechanism, the system architecture of the image processing system can be seen in Figure 5. As shown in Figure 5, the image processing system includes at least a video generation model, input data for the video generation model (i.e., N latent space noise initialization features used when generating the frame sequence), a first image processing module 301, and a second image processing module 302. The composition and principle of the first image processing module 301 can be found in the description of the embodiment shown in Figure 3 above. Similarly, the composition and principle of the second image processing module 302 can be found in the description of the embodiment shown in Figure 4b above, and will not be repeated here.

[0078] Therefore, the innovative dual-stream consistency embedding mechanism designed in this application consists of two parts: a latent space embedding mechanism and a multi-level cross-attention embedding mechanism. These two embedding mechanisms can provide the features extracted from the reference image provided to the video generation model (or the specified information / consistency information used to guide video content generation, etc.) to the latent space noise initialization features used during frame sequence generation (i.e., the latent space created in the early stages of generation), and to each level of the feature extraction module of the Unet network in the video generation model. This not only allows the video generation model to perceive the image content of the reference image from the dimension of the input data, but also allows the video generation model to perceive the image content of the reference image during the video generation process. This design, which allows the video generation model to fully perceive the reference image and its main content information at multiple levels, effectively ensures that the image content of each frame in the generated video sequence maintains a high degree of consistency with the currently provided reference image, making the image content of the entire video sequence more coherent and preventing content abruptness.

[0079] It should be noted that, for ease of explanation, the image processing scheme provided in this application embodiment will be described in detail below using the feature embedding mechanism as a dual-stream consistency embedding mechanism as an example.

[0080] In practical applications, the image processing system described in this application can be deployed on a single computer device or distributed across multiple computer devices. After receiving a reference image and content description text, the computer device can invoke the image processing system to use an image processing scheme to convert the reference image into a target video based on the reference image and content description text. Specifically, the image processing system can be deployed directly on the computer device or within an application program that runs on the computer device. An application program can refer to a computer program designed to perform one or more specific tasks. Classifying applications according to different dimensions (such as the application's operating method, function, etc.) can yield the types of the same application under different dimensions. For example, according to the application's operating method, applications may include, but are not limited to: clients installed on a terminal, small programs that can be used without downloading and installation (as subroutines of the client), and World Wide Web (Web) applications opened through a browser, etc. As another example, according to the application's functional type, applications may include, but are not limited to: instant messaging (IM) applications, content interaction applications, etc. Instant messaging applications refer to internet-based applications that facilitate real-time communication and social interaction. These applications can include, but are not limited to, social applications with communication functions, map applications with social interaction functions, and game applications. Content-interactive applications refer to applications capable of content interaction, such as sharing platforms, personal spaces, news, video, and app store applications.

[0081] The computer equipment may include terminals and servers. Terminals may include, but are not limited to, smartphones (such as smartphones running the Android system or smartphones running the Internetworking Operating System (IOS), tablets, portable personal computers, in-vehicle devices, head-mounted devices, smart TVs, or smart home devices. This application does not limit the type of terminal device; this is only a description of the specific type. A server is the backend server corresponding to the terminal, used to interact with the terminal to provide computing and application service support for the applications deployed on the terminal. A server may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0082] Furthermore, the image processing solution provided in this application embodiment can be applied to any application scenario that requires image-to-video conversion. In such application scenarios, for a video generation task where the user wants video content, a specified reference image is input, and then an entire video frame sequence is generated. The main body of the image content in each video frame in this video frame sequence is consistent with the main body of the image content in the reference image. At the same time, through the control of the feature attenuation mechanism, the appearance of the main body of the image content in each video frame can be different from the appearance of the main body of the image content in the reference image, ensuring that the overall video content of the video frame sequence has a certain degree of diversity. For example, application scenarios may include, but are not limited to: game application scenarios, camera application scenarios, and dynamic emoticon generation scenarios (such as GIF (Graphics Interchange Format) emoticon generation scenarios). The process of implementing the image processing solution in these exemplary application scenarios is briefly introduced below; wherein:

[0083] The application scenario is a game application scenario, specifically the production of game promotional materials. In this scenario, it is often necessary to convert a game promotional image into a game promotional video (such as a game video showing subtle movements of game characters within a promotional image) to better promote the game. In this case, the computer equipment deploying the image processing system can be a server on the game development side. Game developers can input a reference image of a game scene or a reference image containing game characters into the server. The image processing system deployed on the server receives the reference image and uses a first image processing module and / or a second image processing module to perform image processing on the reference image (such as image encoding, duplication, and feature attenuation processing) to obtain N attenuated image encoded features. These N attenuated image encoded features are then embedded into the model inference stage of the video generation model (such as embedding them into the N latent space noise initialization features used when generating the frame sequence or into the feature extraction modules at various levels in the Unet network). This allows the generation of a target video based on the reference image that is highly consistent with the game style, scene, and characters, and can include diverse game scenes or subtle movements of game characters. Furthermore, by varying the attenuation intensity during feature attenuation processing, the rate of change of video content in the generated target video can be controlled, thereby achieving the effect of creating diverse target videos for game promotional materials.

[0084] The application scenario is a camera application scenario, specifically a scenario of converting camera images into video. In this scenario, users may have a need to convert images captured or downloaded from the network into videos. In this case, the computer device is the user's terminal, which runs a camera application. The image processing solution provided in this embodiment is deployed on this camera application. After the user captures a reference image using the camera application on the terminal (or uploads a reference image through the camera application), they can choose to generate a dynamic video live (i.e., a real-time video stream) based on the reference image. At this time, the camera application inputs the reference image into the image processing system, so that the image processing system can convert the reference image according to the image processing solution provided in this embodiment to obtain the target video. The image content of each frame in the target video has both content consistency and content differences with the image content of the reference image, thereby improving the user experience of using the camera application and, to some extent, enhancing the fun of the camera application.

[0085] The application scenario is the generation of animated emoticons, which are GIF animations depicting subtle facial expressions. In this case, the computer device can be a user-held terminal or a server held by the emoticon developer. Using the computer device as the terminal, with an application for creating emoticons deployed on it, the user can upload a reference image, such as a composite image from a rage comic or a famous scene from a movie or TV show. The application then inputs the reference image into an image processing system, which uses an image processing scheme to convert the reference image into a target video. This target video is the animated emoticon, where each frame of the video contains the image of the person in the reference image, and the dynamic effect of multiple frames playing out the subtle movements of the person is captured.

[0086] It should be noted that the above three application scenarios are examples of the application scenarios to which the image processing solution provided in the embodiments of this application is applicable, and do not limit the application scenarios to which the image processing solution is applicable.

[0087] To facilitate understanding of the application scenarios applicable to the aforementioned image processing solutions, an example application scenario for an image processing solution is described below with reference to Figure 6. As shown in Figure 6, the image processing system includes a terminal 601 and a server 602. This embodiment does not limit the number or naming of the terminals and servers. The terminal 601 and server 602 can communicate directly or indirectly via wired or wireless means. This embodiment does not limit the communication methods between the devices. For example, communication methods between devices may include, but are not limited to, HTTP requests, Remote Procedure Calls, sockets, and content sharing.

[0088] The image processing scheme provided in this application embodiment can be executed by a computer device. This computer device carries a video generation model trained during the model training phase of this application embodiment, as well as a first image processing module and / or a second image processing module that jointly implement the image processing scheme with the video generation model. The computer device can be terminal 601 and / or server 602 in the system shown in Figure 6. That is, this application embodiment supports the image processing scheme being executed by either terminal 601 or server 602, or by both terminal 601 and server 602. The following description uses an example where the image processing scheme is jointly executed by terminal 601 and server 602, with terminal 601 deploying the first and second image processing modules, and server 602 deploying the video generation model, to illustrate an image-to-video application scenario to which the image processing scheme is applicable.

[0089] In practice, a user can upload or capture a reference image via terminal 601 and set a content description text. This content description text describes the video content of the target video that the user wants to generate based on the reference image. For example, if the content description text is "a bear doing calisthenics," the user wants the main subject "bear" in the target video generated based on the reference image to perform calisthenics actions. After receiving the reference image, terminal 601 uses a first image processing module to perform image encoding and feature attenuation processing on the reference image, obtaining N attenuated latent space reference features. Simultaneously, it uses a second image processing module to perform image encoding and feature attenuation processing on the reference image, obtaining N attenuated image feature vectors. Terminal 601 transmits the N attenuated latent space reference features, the N attenuated image feature vectors, and the content description text to server 602. After receiving N attenuated latent space reference features, N attenuated image feature vectors, and content description text, server 602 embeds the N attenuated latent space reference features into the N latent space noise initialization features used when generating the frame sequence to obtain N latent space target features. It also performs text recognition processing on the content description text to obtain a text recognition vector. Thus, during the image-to-video conversion process using the text recognition vector and the N latent space target features, server 602 also embeds the N attenuated image encoding feature vectors into the feature extraction modules at each layer of the Unet network in the video generation model to ultimately obtain the target video. In the target video, the main subject of each frame's image content is a "bear," which is consistent with the "bear" in the reference image content. The difference lies in the bear's appearance in the target video; its posture and expression are consistent with the content description text but differ from the bear's appearance in the reference image content. To facilitate users' timely access to the converted target video, server 602 will also return the generated target video to terminal 601, so that users can browse, copy, download, and forward the target video from terminal 601.

[0090] Based on the above brief introduction to the image processing scheme and application architecture provided in the embodiments of this application, the following points should also be noted:

[0091] ① The application architecture shown in Figure 6 mentioned above in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. Those skilled in the art will understand that with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems. That is to say, the schematic diagram of the image-to-video application architecture shown in Figure 6 is an exemplary architecture schematic diagram; in actual applications, the number and distribution of devices (such as terminals and servers) included in this architecture can change, and the embodiments of this application do not limit the schematic diagram of the image-to-video application architecture. All embodiments recorded in this application, unless there are technically mutually exclusive situations, can be freely combined to obtain further embodiments. Such combined embodiments should be regarded as specifically recorded in this application.

[0092] ②The above example illustrates the image processing scheme's workflow, using the deployment of the first and second image processing modules on terminal 601 and the trained video generation model on server 602 as an example. In practical applications, the workflow of the image processing scheme implemented in the architecture shown in Figure 6 varies depending on the deployment locations of the first image processing module, the second image processing module, and the video generation model. For example, if the first image processing module, the second image processing module, and the video generation model are all deployed on the server, then after receiving the reference image and content description text uploaded by the user, terminal 601 can directly transmit the reference image and content description text to server 602, where server 602 uses the first and second image processing modules to perform image encoding and feature attenuation processing on the reference image. Furthermore, if only one feature embedding mechanism is used, terminal 601 or server 602 may only include the first image processing module or the second image processing module.

[0093] ③ The data collection and processing in this application embodiment should strictly comply with the requirements of relevant laws and regulations. Obtaining personal information requires the knowledge or consent of the individual (or a legal basis for information acquisition), and subsequent data use and processing should be carried out within the scope of laws, regulations, and the authorization of the personal information subject. For example, when this application embodiment is applied to specific products or technologies, if a reference image is uploaded, permission or consent from a user with the authority to handle the reference image is required. Furthermore, the collection, use, and processing of related data (such as the conversion of reference images) must comply with relevant national and regional laws, regulations, and standards.

[0094] Please refer to Figure 7, which is a flowchart illustrating an image processing method provided in an exemplary embodiment of this application. This flowchart primarily illustrates the specific implementation process of converting an image to video using a video generation model and a first image processing module, from the perspective of model reasoning. The method flow shown in Figure 7 can be executed by the aforementioned computer device. Taking the terminal 601 shown in Figure 6 as an example, the image processing method may include, but is not limited to, steps S701-S704:

[0095] S701: Obtain the reference image and content description text.

[0096] As described above, a reference image, referred to as a ref image in this embodiment, is an image whose content serves as the base content for the video content of the target video to be generated. Introducing a reference image during video generation allows the reference image content to guide the generation of image content for each video frame in the frame sequence, ensuring that the main subject (such as the main character and environment) of the image content in each frame of the generated target video is consistent with the main subject of the reference image content; that is, there is content consistency between the reference image content and the image content of each video frame in the frame sequence. Similarly, content description text is a string used to describe the video content that the target video to be generated should contain. Introducing content description text during video generation allows for detailed guidance on the image content of each video frame in the target video, such as the content details of the video frame's image content or the content direction of the entire frame sequence. It should be noted that both the reference image and the content description text are used to generate the same target video; therefore, it is necessary to ensure that there is no semantic conflict between the image content of the reference image and the image content described by the content description text; that is, the main content of the image content described by the content description text needs to be consistent with the main content of the image content of the reference image. For example, if the main subject of the reference image is a standing bear, and the content description text is "dancing girl," then there will be a semantic conflict between the main subject of the reference image, "bear," and the main subject of the image content described in the content description text, "girl."

[0097] Therefore, in the process of generating a target video using reference images and content description text, the role of the reference image is to ensure content consistency between the main content of each frame in the generated target video and the image content of the reference image. The role of the content description text is to ensure that the presentation and other aspects of the main content of each frame in the generated target video match the image content described in the content description text. This effectively ensures that each frame in the target video is generated based on the image content of the reference image, and that the content description text guides the generation of differences between the image content of each frame in the target video and the image content of the reference image, thus ensuring that the overall content changes of the target video conform to the image content described in the content description text.

[0098] Depending on the application scenario of the image processing method provided in this application embodiment, the providers of the reference image and content description text differ. For example, the provider can be a user or a developer, and the number of providers can be one or more. For instance, assuming the image processing method is applied to a camera application scenario, in this scenario, the user needs to convert images to video. Therefore, the user needs to upload a reference image and input content description text via a computer device. Exemplarily, the camera application scenario is provided by a camera app deployed on a terminal. As shown in Figure 8, a page in the camera app can have an image-to-video button, such as button 801. When button 801 is triggered, an upload page 802 is displayed. This upload page 802 includes an image upload area 8021 and a text upload area 8022. Thus, the user can upload historically captured or downloaded images as reference images in the image upload area 8021, and manually input or copy and paste text as content description text in the text upload area 8022. This application embodiment does not limit the source and acquisition method of the reference image and content description text; this is only a description of the specific implementation.

[0099] S702: Encode and copy the reference image to obtain N image encoding features to be attenuated.

[0100] In order to use the image content of a reference image to guide the generation of image content for N video frames in a target video, this application embodiment supports encoding and copying the reference image after obtaining the reference image, to obtain N image encoding features to be attenuated extracted from the reference image. These N image encoding features to be attenuated are all the same, and any one of the image encoding features to be attenuated can be understood as the image features of the reference image (such as color features, texture features, shape features, and spatial relationship features, etc.), which can be used to characterize the content characteristics of the reference image content.

[0101] In the specific implementation, after obtaining the reference image, image encoding processing is first performed on the reference image to obtain the image encoding features to be attenuated. The purpose is to extract the content features or content characteristics of the reference image to represent its content. Then, considering that the target video to be generated is a frame sequence composed of N video frames, in order to facilitate the generation of the image content of each video frame based on the image content of the reference image, the image encoding features to be attenuated need to be copied N times to obtain N image encoding features to be attenuated.

[0102] As mentioned above, this application provides two feature embedding mechanisms: latent space embedding and multi-level cross-attention embedding. Different feature embedding mechanisms correspond to different image encoding features to be attenuated. In other words, different feature embedding mechanisms require different image encoding processing methods to encode the reference image, ensuring that the image encoding features extracted from the reference image can be embedded into the video generation model.

[0103] In this embodiment, the image encoding processing method corresponding to the latent latent space embedding mechanism is described. When the feature embedding mechanism is the latent latent space embedding mechanism, the image encoding feature to be attenuated corresponding to the latent latent space embedding mechanism can be called the latent reference feature to be attenuated, which can be represented as ref latent or ref feature. When the feature embedding mechanism is the latent latent space embedding mechanism, the system framework diagram provided in this embodiment can be seen in Figure 3 above. As shown in Figure 3, the first image processing module 301 included in the system framework includes the encoder part of the variational autoencoder built into the video generation model (Stable Diffusion model), namely the VAE encoder. Thus, the variational autoencoder built into the video generation model can be used to perform image encoding processing on the reference image, aiming to encode the reference image into the latent reference feature to be attenuated required by the video generation model. The principle of the variational autoencoder performing image encoding processing on the reference image can include: the variational autoencoder receives the input reference image and extracts the image features of the reference image through a series of convolutional layers or other feature extraction algorithms. These image features can be used to characterize key information in the reference image. The variational autoencoder then maps the image features of the reference image to the latent space, obtaining the latent space reference features to be attenuated that characterize the content features of the reference image in the latent space. By mapping the image features of the reference image to the latent space, more latent features of the reference image can be learned using these image features, so as to deepen the understanding of the reference image. This facilitates the subsequent generation of video content based on more accurate and richer image features, thereby improving the quality of the video content.

[0104] Furthermore, the variational autoencoder in the system framework shown in Figure 3 is used to perform image encoding processing on the reference image to obtain the latent space reference features to be attenuated. Then, the latent space reference features to be attenuated are directly copied into N copies, resulting in N latent space reference features to be attenuated. Each latent space reference feature to be attenuated corresponds to a video frame in the target video to be generated, thereby ensuring that the video content of each video frame in the target video is consistent with the image content of the reference image. Moreover, after obtaining the image encoding features to be attenuated by coarse-grained image encoding of the reference image, it is only necessary to copy the image encoding features to be attenuated to obtain N image encoding features to be attenuated, thereby improving the generation efficiency of N image encoding features to be attenuated and reducing the generation cost.

[0105] It is worth noting that this embodiment of the application chooses to use the variational autoencoder built into the video generation model to perform image encoding processing on the reference image, mainly based on the following considerations: ① As shown in the system framework of the Stable Diffusion model during the model training phase in Figure 1a, the massive sample data used during the model training phase all output images to the variational autoencoder, specifically the VAE encoder, which encodes the data and then adds noise. Therefore, this embodiment of the application supports using the VAE encoder built into the video generation model to perform image encoding processing on the reference image, which can ensure that the feature semantics extracted from the reference image are more adaptable to the training semantics of the Stable Diffusion model.

[0106] ② Considering that the features calculated for the reference image (i.e., the aforementioned latent space reference features to be attenuated, specifically the attenuated latent space reference features after feature attenuation processing) need to be embedded into the Stable Diffusion model trained as mentioned in ①, specifically into the latent space noise latent used when generating frame sequences using the Stable Diffusion model, it is necessary to ensure that the feature size of the latent space reference features to be attenuated extracted from the reference image is consistent with the feature size of the latent space noise latent. Therefore, this embodiment uses the variational autoencoder trained by the Stable Diffusion model during the model training phase to perform image encoding processing on the reference image. This ensures that the feature size of the latent space reference features to be attenuated extracted from the reference image is consistent with the feature size of the latent space noise latent, that is, it ensures that the latent space reference features to be attenuated obtained by image encoding processing of the reference image during the model inference phase can meet the size requirements of the Stable Diffusion model for input data.

[0107] ③ As described above, the Stable Diffusion model trains its encoder (vae encoder) using a massive number of images during the model training phase. Therefore, the well-trained encoder (vae encoder) is able to completely extract the main semantic content of the image. Thus, in this embodiment, the use of the encoder (vae encoder) trained by the Stable Diffusion model during the model training phase to encode the reference image further enhances the extraction of the main semantic content of the reference image and improves the accuracy of feature extraction.

[0108] S703: Perform feature attenuation processing on N image coding features to be attenuated to obtain N attenuated image coding features.

[0109] Based on the foregoing description, image encoding processing is performed on the reference image to obtain N image encoding features to be attenuated. If these N identical image encoding features are directly used to guide the generation of image content for N video frames in the target video, the generated N video frames will have identical image content. For example, if N=16, and the generation of all 16 video frames uses the original consistency information (i.e., image encoding features to be attenuated) extracted from the reference image, then all 16 video frames generated using the original image encoding features to be attenuated will be completely identical. However, in reality, according to the video playback order, the image content of adjacent video frames in the N video frames of the target video has content differences, thus ensuring that the entire video content is dynamically changing, such as changes in a person's posture or changes in the scene.

[0110] To achieve flexible adjustment of image content for N video frames, this application supports using a feature attenuation strategy to perform feature attenuation processing on N image coding features extracted from a reference image, and using the N attenuated image coding features to guide the generation of N video frames. The feature attenuation strategy can be understood as a rule that progressively weakens N identical image coding features from a feature dimension, specifically including one or more rule parameters. Thus, after feature attenuation processing, the resulting N attenuated image coding features are not identical, and the feature differences between the N attenuated image coding features and the original image coding features increase. Therefore, when using the N attenuated image coding features to guide the generation of N video frames, it can be ensured that the image content of the N video frames differs from the image content of the reference image.

[0111] The main principle of the feature attenuation strategy provided in this application embodiment is as follows: A corresponding attenuation coefficient is set for each of the N video frames, and the attenuation coefficient is used to perform feature attenuation processing on the image coding features to be attenuated referenced during the generation of the corresponding video frame. Specifically, it supports pre-setting attenuation parameters, the value of which is a decimal between (0, 1). Therefore, the smaller the value of the attenuation parameter, the greater the content difference between adjacent video frames, thus making the video content of the target video change faster. Thus, the attenuation parameter can be used to indicate the degree of content difference between the image content of adjacent video frames in a frame sequence. Then, N attenuation coefficients are calculated using the pre-set attenuation parameters, and the N attenuation coefficients correspond one-to-one with the N image coding features to be attenuated; the one-to-one correspondence means that each attenuation coefficient is used as a feature weight of an image coding feature to be attenuated. Finally, the N attenuation coefficients are multiplied by their corresponding image coding features to be attenuated to obtain N attenuated image coding features.

[0112] To facilitate understanding, the feature attenuation process described above will be explained in detail below with reference to Figure 9. As shown in Figure 9, assuming the attenuation parameter is represented by w, the attenuation coefficient corresponding to the first image coding feature to be attenuated in the N image coding features to be attenuated is 1.0 by default, and the attenuation coefficient corresponding to the second image coding feature to be attenuated is 1.0 × w (i.e., w). 1 The attenuation coefficient corresponding to the third image encoding feature to be attenuated is 1.0×w×w (i.e., w). 2 The attenuation coefficient corresponding to the fourth image encoding feature to be attenuated is 1.0×w×w×w (i.e., w). 3 ), and so on, the attenuation coefficient corresponding to the Nth image encoding feature to be attenuated is 1.0×w N-1 (i.e. w) N-1 Then, the first image coding feature to be attenuated among the N image coding features to be attenuated is multiplied by its corresponding attenuation coefficient 1.0 to obtain the first attenuated image coding feature. This first attenuated image coding feature is the same as the first image coding feature to be attenuated, that is, the image content of the first video frame in the N video frames is generated entirely based on the image content of the reference image. The second image coding feature to be attenuated is multiplied by its corresponding attenuation coefficient w1 to obtain the second attenuated image coding feature. Since w is a decimal in the range of 0 to 1, this second attenuated image coding feature is different from the second image coding feature to be attenuated. The second attenuated image coding feature attenuates the semantic information of the content of the second image coding feature to be attenuated, that is, the image content of the second video frame in the N video frames is generated only based on a portion of the image content of the reference image. And so on, calculating the attenuated image coding feature corresponding to each of the N image coding features to be attenuated.

[0113] It should be noted that: ① This application does not limit the specific value of the attenuation parameter; depending on the value of the attenuation parameter, the rate of content change between frames varies. For example, the larger the value of the attenuation parameter, the greater the rate of content change between frames, meaning the greater the content difference between the image content of the video frame and the image content of the reference image. Thus, by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), the rate of content change in the final generated target video can be effectively controlled, enriching the diversity of content changes in the video frame sequence while meeting the flexible control requirements for inter-frame content changes in different scenarios.

[0114] ② This application does not limit the calculation rules for the attenuation coefficient based on the attenuation parameter in the embodiments. As shown in Figure 9 above, the calculation rule is to use the power of the attenuation parameter as the attenuation coefficient. In practical applications, the calculation rule can also include the square root of the attenuation coefficient, etc. For example, assuming the attenuation parameter is a positive number greater than 1, the default attenuation coefficient corresponding to the first image coding feature to be attenuated in the N image coding features to be attenuated is 1.0, and the attenuation coefficient corresponding to the second image coding feature to be attenuated is 1.0 × w. 1 / 2 The attenuation coefficient corresponding to the third image encoding feature to be attenuated is 1.0×w. 1 / 2 ×w 1 / 2 And so on, the attenuation coefficient corresponding to the Nth image encoding feature to be attenuated.

[0115] S704: Based on the content description text, perform video generation processing on N attenuated image coding features to generate the target video.

[0116] After calculating N attenuated image coding features, these features are embedded into the video generation process. This allows the generation of the target video content to be guided by the image content of the reference image, ensuring consistency between the image content of the reference image and the image content of each frame in the target video. Specifically, the computer device acquires N latent noise initialization features, which are obtained by adding random noise to a blank latent reference feature. In short, the latent noise initialization feature can be understood as being constructed by adding random noise to a blank image, facilitating the processing of the random noise in the latent noise initialization feature to draw video frames. Considering that one latent noise initialization feature is used to generate one video frame in the frame sequence, the latent noise initialization feature needs to be copied to obtain N latent noise initialization features. Furthermore, the computer device calls the text encoder module to perform text recognition processing on the content description text, obtaining a text recognition vector. This text recognition vector is used to represent the semantic information of the content description text. Furthermore, the video generation model is invoked to perform video generation processing on the text recognition vector and N latent space noise initialization features, and N attenuated image coding features are embedded in the video generation process to obtain the target video.

[0117] As described above, this application provides two feature embedding mechanisms: latent space feature embedding and multi-level cross-attention embedding. The following section, using the system framework shown in Figure 3, describes the specific implementation process of using the latent space feature embedding mechanism. Specifically, it describes calling a video generation model to initialize the text recognition vector and N latent space noise features, and embedding N attenuated image-coded features during the video generation process to obtain the target video. As shown in Figure 3, the image-coded features to be attenuated are represented as the latent space reference features to be attenuated, and the attenuated image-coded features are represented as the attenuated latent space reference features. In this case, before inputting the N latent space noise initialization features into the video generation model, the N attenuated latent space reference features are first embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set. This latent space reference feature set includes N latent space target features, where each latent space target feature is obtained by embedding an attenuated latent space reference feature into its corresponding latent space noise initialization feature. In other words, the N attenuated latent space reference features are embedded with their corresponding latent space noise initialization features from the N latent space noise initialization features to obtain the latent space reference feature set. Thus, when the video generation model is called to process the latent space reference feature set and the text recognition vector for video generation, the consistency information of the reference image can be embedded into the generation process of each video frame during frame sequence generation. This ensures that the image content of each video frame in the resulting target video references the reference image content of the reference image, thereby guaranteeing that the image content of each video frame in the final generated target video matches the reference image content of the reference image. For example, the main subject of the video frame's image content is the same as the reference image content of the reference image, such as the subject's style, shape, and action being the same or similar.

[0118] The aforementioned embedding of N attenuated latent space reference features into corresponding latent space noise initialization features refers to performing point-to-point addition operations on the N attenuated latent space reference features and their corresponding latent space noise initialization features. In other words, performing point-to-point addition operations on the N attenuated image coding features and their corresponding latent space noise initialization features yields a set of latent space reference features. This set of latent space reference features includes N latent space target features obtained from the point-to-point addition operations. Each latent space target feature is obtained by performing point-to-point addition operations on the attenuated image coding features and their corresponding latent space noise initialization features. For example, a schematic diagram of performing point-to-point addition operations on the N attenuated latent space reference features and their corresponding latent space noise initialization features can be seen in Figure 10. As shown in Figure 10, performing point-to-point addition operations on the attenuated image coding features and their corresponding latent space noise initialization features means performing addition operations on the pixels at the same position in the attenuated image coding features with the same feature size and the corresponding latent space noise initialization features to generate new latent space target features.

[0119] In summary, on the one hand, it supports feature attenuation processing of latent space reference features extracted from reference images, helping the video generation model gradually reduce its reliance on reference images during the generation of N video frames. This allows for greater incorporation of image content described in the content description text to control the dynamic changes in video content. Controlling the attenuation coefficient enables greater diversity in video content. On the other hand, it supports effectively controlling the rate of change of the final generated target video content by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), thereby enriching the diversity of content changes in the video frame sequence. On the other hand, it supports embedding consistency information extracted from reference images into the video generation process. When the feature embedding mechanism is a latent space embedding mechanism, the attenuated latent space reference features extracted from the reference images can be embedded into the latent space noise initialization features used when generating the frame sequence before video generation. This enables the direct transfer of the main content semantic information in the reference image to the generation process of the image content of each video frame when generating the frame sequence, which guides the generation of the image content of each video frame and ensures the specific content consistency between the image content of the video frame and the image content of the reference image. Furthermore, the image content described by the content description text guides the content differences between the image content of the video frame and the image content of the reference image, thereby helping users obtain a target video that matches the image content described by the content description text and has content consistency with the image content of the reference image.

[0120] Please refer to Figure 11, which is a flowchart illustrating another image processing method provided by an exemplary embodiment of this application. This flowchart mainly illustrates the specific implementation process of converting an image to video using a video generation model and a second image processing module from the perspective of model reasoning. The method flow shown in Figure 11 can be executed by the aforementioned computer device. Taking the terminal 601 shown in Figure 6 as an example, the image processing method may include, but is not limited to, steps S1101-S1106:

[0121] S1101: Obtain the reference image and content description text.

[0122] It should be noted that the specific implementation process of the embodiment shown in step S1101 is the same as the specific implementation process shown in step S701 of the embodiment shown in Figure 7 above. Please refer to the relevant description of the specific implementation process shown in step S701 above, which will not be repeated here.

[0123] S1102: Perform image coding processing on the reference image to obtain the image coding features to be attenuated.

[0124] S1103: Copy the image coding features to be attenuated N times to obtain N image coding features to be attenuated.

[0125] In steps S1102-S1103, as described above, the feature embedding mechanism provided in this application embodiment includes: a latent space embedding mechanism and a multi-level cross-attention embedding mechanism. When the feature embedding mechanism is a latent space embedding mechanism, the specific implementation process shown in steps S1102-S1103 can be found in the relevant description of the relevant paragraph shown in step S702 of the embodiment shown in Figure 7 above, and will not be repeated here. In this application embodiment, the specific implementation process of image copying processing for the reference image (i.e., the specific implementation process shown in steps S1102-S1103) is mainly described when the feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0126] Specifically, when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the specific implementation process shown in steps S1102-S1103 is executed by the second image processing module shown in Figure 4b. This second image processing module includes an image encoder and a feature attenuation submodule. The image encoder performs image encoding processing on the reference image to obtain the encoded features of the image to be attenuated, specifically the image feature vector to be attenuated (represented as image embedding). The feature attenuation submodule performs feature attenuation processing on the image feature vector encoded by the image encoder to obtain the attenuated image feature vector. Furthermore, the attenuated image feature vector output by the feature attenuation submodule is input into each level of the feature extraction module in the Unet network, allowing the entire Unet network to perceive the semantic information of the main content in the reference image at multiple levels, thereby improving the consistency of the image content of the reference image with the image content of each video frame in the frame sequence.

[0127] In this embodiment of the application, the image encoder used in the multi-level cross-attention embedding mechanism performs image encoding processing on the reference image, as shown in Figure 12. As shown in Figure 12, the process of the second image processing module performing image encoding processing on the reference image to obtain the feature vector of the image to be attenuated, and copying the feature vector of the image to be attenuated, may include, but is not limited to, steps s11-s13; wherein:

[0128] s11: Perform image preprocessing on the reference image to obtain the initial image feature vector.

[0129] As shown in Figure 13, the image encoder included in the second image processing module includes a preprocessing encoding module, which can be represented as a clip image encoder (abbreviated as clip model). As described above, the clip model can fully recognize elements such as objects, scenes, and actions in the image, and can also understand text related to the image, such as tags, descriptions, and titles. Therefore, by setting this clip model in the image encoder in this embodiment, the better recognition performance of the clip model can achieve full and accurate recognition of the semantic information of the content in the reference image.

[0130] Specifically, after the reference image is output to the image encoder, the preprocessing encoding module in the image encoder first performs image preprocessing on the reference image to obtain an initial image feature vector. This initial image feature vector contains the image feature vector to be attenuated and preprocessing information; among them, the image feature vector to be attenuated is used to characterize the main semantic information of the reference image, and the preprocessing information indicates the image analysis information for the reference image. The image analysis information mainly includes the semantic information of the clip model itself, such as the classification prediction information of the clip model for the content in the reference image.

[0131] s12: Extract the feature vector of the image to be attenuated from the initial image feature vector. Specifically, perform content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated.

[0132] After extracting the initial image feature vector from the reference image based on the aforementioned step s11, content extraction processing (or content purification processing) needs to be performed on the initial image feature vector to extract the image feature vector to be attenuated, which represents the main semantic information of the reference image. In this embodiment, a preset initial image semantic vector is used to assist in the content purification of the initial image feature vector; this initial image semantic vector can be represented as an initial image embedding, which can be used to indicate the weight distribution for content extraction processing from the initial image semantic vector, and the initial image semantic vector can be initialized using random weights. Specifically, after obtaining the preset initial image semantic vector, content extraction processing is performed on the initial image feature vector based on the initial image semantic vector to obtain the image feature vector to be attenuated. Referring to Figure 13, the content extraction process here includes attention operations and fully connected operations. Specifically, the preset initial image semantic vector and the initial image feature vector encoded by the clip image encoder are input together into the cross attention layer for attention operations. Then, the cross attention layer is input into a fully connected layer (Multilayer Perceptron, MLP) to integrate the features obtained by the cross attention layer, so as to purify the image feature vector to be attenuated from the initial image feature vector.

[0133] Furthermore, this application embodiment sets up a multiple content purification mechanism to achieve multiple content extraction processes on the initial image feature vector, aiming to improve the feature purity of the image feature vector to be attenuated extracted from the initial image feature vector. As shown in Figure 13, the content extraction process is performed by a content extraction module. It is assumed that the number of content extraction modules is M, where M is a positive integer; one content extraction module corresponds to one content extraction / purification process, and each content extraction module includes a cross-attention layer and a fully connected layer. Based on this, after the first content extraction module performs content extraction processing on the initial image feature vector and the initial image semantic vector, the output information of the first content extraction module can be input into the second content extraction module to continue content extraction processing, and so on, with the output information of the last content extraction module serving as the image feature vector to be attenuated.

[0134] In the aforementioned multiple content extraction processes, to ensure that the original semantic content of the reference image is not significantly lost during these processes, this embodiment of the application further designs a method where the input information of the cross-attention layer within the same content extraction module is added to the output information of the fully connected layer, serving as the input information for the next content extraction module. This ensures that the original semantic content of the reference image is not significantly lost in each content extraction process. As shown in Figure 13, assuming the number of content extraction modules is M, when M=1, the input information of the first content extraction module, i.e., the input information processed in the first content extraction process, is: the initial image semantic vector and the initial image feature vector. When M=2, the input information of the second content extraction module, i.e., the input information processed in the second content extraction process, is: the output information of the first content extraction module (i.e., the processing result obtained from the first content extraction process) and the concatenated information of the initial image semantic vector (i.e., the information obtained by concatenating features or vectors). When M > 2, the input information of the Mth content extraction module, that is, the input information processed by the Mth content extraction process, is the concatenation of the output information of the (M-1)th content extraction module and the output information of the (M-2)th content extraction module.

[0135] s13: Copy the feature vector of the image to be attenuated N times to obtain N feature vectors of the image to be attenuated.

[0136] After extracting the clean image feature vector to be attenuated from the initial image feature vector, the image feature vector to be attenuated needs to be copied into N copies according to the number of frames N of the target video to be generated, resulting in N image feature vectors to be attenuated.

[0137] In summary, based on the specific implementation process shown in steps s11-s13 above, by performing image preprocessing on the reference image using the knuckles, it can be ensured that the initial image feature vector is a feature vector that can accurately represent the semantics of the reference image. Therefore, the feature vector to be attenuated extracted from this initial image feature vector is also a feature vector that can accurately represent the semantics of the reference image. Consequently, generating image content for video frames based on this feature vector to be attenuated ensures content consistency between the video frames and the reference image. In this embodiment, under the multi-level cross-attention embedding mechanism, the consideration for using an image encoder to encode the reference image is to ensure that the feature vector to be attenuated encoded by the image encoder can be embedded into the feature extraction module at each level of the Unet network in the video generation model. That is, the data format of the feature vector to be attenuated after image encoding using the image encoder conforms to the data format requirements of the feature extraction module for the input data.

[0138] S1104: Perform feature attenuation processing on N image coding features to be attenuated to obtain N attenuated image coding features.

[0139] It should be noted that the specific implementation process shown in step S1104 is the same as the specific implementation process shown in step S703 in the embodiment shown in Figure 7. Please refer to the relevant description of the specific implementation process shown in step S703, which will not be repeated here.

[0140] S1105: Obtain N latent space noise initialization features and perform text recognition processing on the content description text to obtain the text recognition vector.

[0141] S1106: Call the video generation model to perform video generation processing on the text recognition vector and N latent space noise initialization features, and embed N attenuated image coding features during the video generation process to obtain the target video.

[0142] In steps S1105-S1106, when the feature embedding mechanism is a latent space embedding mechanism, the specific implementation process shown in steps S1105-S1106 can be found in the relevant description of the relevant paragraph shown in step S704 of the embodiment shown in Figure 7 above, and will not be repeated here. In this embodiment, the specific implementation process of embedding the attenuated image feature vector into the video generation model (i.e., the specific implementation process shown in steps S1105-S1106) is mainly described when the feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0143] Specifically, when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the specific implementation process shown in steps S1105-S1106 can be found in Figure 14. As shown in Figure 14, this embodiment of the application fine-tunes the feature extraction module (Unet block) in the Unet network to obtain a fine-tuned feature extraction module (new Unet block). Specifically, the fine-tuned feature extraction module adds a text-based cross-attention mechanism. Thus, the fine-tuned feature extraction module includes a first attention submodule corresponding to text and a second attention submodule corresponding to images. That is, compared to the feature extraction module before fine-tuning shown in Figure 1b, the fine-tuned feature extraction module mainly adds a second attention submodule (this module integrates cross attention). This second attention submodule is used to receive the attenuated image feature vector extracted by the second image processing module for the reference image, and after performing attention operations on the attenuated image feature vector, it adds the attention operation result to the attention result obtained by performing attention operations on the text recognition vector, that is, it can input the main content information in the reference image into the entire video generation process. It should be noted that, when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the feature extraction module mentioned in the embodiments of this application refers to the fine-tuned feature extraction module, namely the feature extraction module shown in Figure 14.

[0144] Based on the fine-tuned feature extraction module shown in Figure 14, the embedding method of the N attenuated image feature vectors output by the second image processing module into the video generation process can include: during the video generation process, embedding the N attenuated image feature vectors into the feature extraction module of each layer in the Unet network of the video generation model; then, calling the feature extraction module to perform video generation processing on the embedded N attenuated image feature vectors, text recognition vectors, and N latent space noise initialization features to obtain the target video. Taking any layer of the multi-level feature extraction module as an example, the process of embedding the N attenuated image feature vectors into the target feature extraction module is described. Specifically: through the second attention mechanism in the target feature extraction module, attention operations are performed on the received N attenuated image feature vectors to obtain the second attention operation result; and through the first attention mechanism in the target feature extraction module, attention operations are performed on the received text recognition vectors to obtain the first attention operation result; then, the second attention operation result and the first attention operation result are fused to obtain the output information of the target feature extraction module. Through the above embedding process, the main content information of the reference image can be embedded into the video generation process, specifically into the feature extraction module at each level, so as to improve the content consistency between the image content of each video frame in the final generated target video and the image content of the reference image.

[0145] More specifically, referring to Figure 14, when the target feature extraction module receives N attenuated image feature vectors, it also receives text recognition vectors obtained from the content description text recognition, as well as preceding features. For example, the preceding features received by the first feature extraction module are N latent space noise initialization features. The preceding features received by the other M feature extraction modules (excluding the first feature extraction module) are the feature information after feature extraction processing of the N latent space noise initialization features by the preceding feature extraction module adjacent to that module. Therefore, the process of the target feature extraction module using the second attention submodule to perform attention operations on the N attenuated image feature vectors specifically includes: using the second attention submodule to perform attention operations on the N attenuated image feature vectors and preceding features to obtain the second attention operation result. Similarly, the process of the target feature extraction module using the first attention submodule to perform attention operations on the text recognition vector specifically includes: using the first attention submodule to perform attention operations on the text recognition vector and preceding features to obtain the first attention operation result.

[0146] In summary, on the one hand, it supports feature attenuation processing of latent space reference features extracted from reference images, helping the video generation model gradually reduce its reliance on reference images during the generation of N video frames. This allows for greater incorporation of image content described in the content description text to control the dynamic changes in video content. Controlling the attenuation coefficient enables greater diversity in video content. On the other hand, by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), the speed of content change in the final generated target video can be effectively controlled, thereby enriching the diversity of content changes in the video frame sequence. On the other hand, it supports embedding consistency information extracted from reference images into the video generation process. When the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the attenuated image feature vectors extracted from the reference images can be embedded into the attention operation process of each layer of the feature extraction module in the Unet network. This allows the image content of each video frame generated by the Unet model to match the main semantic information of the reference image, ensuring a high degree of content consistency between the video content of the entire video frame sequence and the image content of the reference image. This ensures that the video content of the entire video frame sequence is more coherent and does not produce abrupt content. Furthermore, the image content described in the content description text guides the content differences between the image content of the video frames and the image content of the reference image, thereby helping users obtain target videos that match the image content described in the content description text and have content consistency with the image content of the reference image.

[0147] Please refer to Figure 15, which is a flowchart illustrating another image processing method provided by an exemplary embodiment of this application. This flowchart primarily illustrates the specific implementation process of converting an image to video using a video generation model, a first image processing module, and a second image processing module, from the perspective of model reasoning. The method flow shown in Figure 15 can be executed by the aforementioned computer device, such as the terminal 601 and / or server 602 shown in Figure 6. The image processing method may include, but is not limited to, steps S1501-S1507:

[0148] S1501: Obtain the reference image and content description text.

[0149] It should be noted that the specific implementation process shown in step S1501 is the same as the specific implementation process shown in step S701 in the embodiment shown in Figure 7 above. Please refer to the relevant description of the specific implementation process shown in step S701 above, which will not be repeated here.

[0150] S1502: Map the reference image to the latent space to obtain the latent space reference features to be attenuated, which characterize the content properties of the reference image in the latent space.

[0151] It should be noted that the specific implementation process shown in step S1502 can be found in the relevant description of the relevant paragraph shown in step S702 in the embodiment shown in Figure 7 above, and will not be repeated here.

[0152] S1503: Perform image preprocessing on the reference image to obtain an initial image feature vector, and perform content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated.

[0153] It should be noted that the specific implementation process shown in step S1503 can be found in the relevant descriptions of the relevant paragraphs shown in steps s11-s12 in the embodiment shown in Figure 12 above, and will not be repeated here.

[0154] The order of execution of steps S1502 and S1503 is not limited in this embodiment.

[0155] S1504: Copy the image coding features to be attenuated N times to obtain N image coding features to be attenuated. The N image coding features to be attenuated include the latent space reference features to be attenuated and the image feature vector to be attenuated.

[0156] It should be noted that the specific implementation process shown in step S1504 is the same as the specific implementation process shown in step S703 in the embodiment shown in Figure 7 above. Please refer to the relevant description of the specific implementation process shown in step S703, which will not be repeated here.

[0157] S1505: Embed the N attenuated latent space reference features into the corresponding latent space noise initialization features to obtain the latent space reference feature set, and input the latent space reference feature set into the video generation model.

[0158] It should be noted that the specific implementation process shown in step S1505 can be found in the relevant description of the relevant paragraph shown in step S704 in the embodiment shown in Figure 7 above, and will not be repeated here.

[0159] S1506: Embed N attenuated image feature vectors into the feature extraction module of each level.

[0160] It should be noted that the specific implementation process shown in step S1506 can be found in the relevant description of the relevant paragraphs shown in step S1106 in the embodiment shown in Figure 11 above, and will not be repeated here.

[0161] The order in which steps S1505 and S1506 are executed is not limited in this embodiment.

[0162] S1507: Generate a target video based on N embedded attenuated image coding features. The N image coding features to be attenuated include the latent space reference features to be attenuated and the feature vector of the image to be attenuated.

[0163] In specific implementation, when the feature embedding mechanism includes both latent latent space embedding and multi-level cross-attention embedding, generating the target video based on the embedded N attenuated image coding features can include: calling the video generation model to perform video generation processing on the text recognition vector, the latent space reference feature set, and the N attenuated image feature vectors to obtain the target video. Therefore, when the feature embedding mechanism includes both latent latent space embedding and multi-level cross-attention embedding, the generation process of the target video will simultaneously reference the N attenuated latent space reference features output by the first image processing module and the N attenuated image feature vectors output by the second image processing module. The embedding method of embedding the N attenuated latent space reference features and the N attenuated image feature vectors into the video generation model can be found in the aforementioned description and will not be repeated here.

[0164] In summary, on the one hand, the embodiments of this application support feature attenuation processing of the latent space reference features extracted from the reference image, helping the video generation model gradually reduce the reference to the reference image during the generation of N video frames, and enabling the incorporation of more image content described by the content description text to control the dynamic changes of the video content. Controlling the attenuation coefficient enables the diversity of video content. On the other hand, the embodiments of this application support controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), which can effectively control the rate of change of the video content in the final generated target video, thereby enriching the diversity of content changes in the video frame sequence. Furthermore, the embodiments of this application simultaneously support two feature embedding mechanisms: the latent space embedding mechanism directly transmits the main semantic information of the reference image to the generation process of the image content of each video frame during the generation of the frame sequence, guiding the generation of the image content of each video frame. The multi-level cross-attention embedding mechanism embeds the attenuated image feature vector extracted from the reference image into the attention operation process of each layer of the Unet network's feature extraction module. This ensures that the image content of each video frame generated by the Unet model matches the main semantic information of the reference image (i.e., the image content of the video frame is generated with reference to the main semantic information of the reference image), achieving a high degree of content consistency between the video content of the entire video frame sequence and the image content of the reference image. This ensures a more coherent video content sequence without abrupt changes. Therefore, by combining these two feature embedding mechanisms, the video generation model can fully receive the main semantic information of the reference image, improving the content consistency between the image content of each video frame in the final target video and the image content of the reference image. Furthermore, the image content described in the content description text guides the content differences between the image content of the video frames and the image content of the reference image, helping users obtain target videos that match both the image content described in the content description text and the image content of the reference image.

[0165] The methods of the embodiments of this application have been described in detail above. To facilitate better implementation of the above-described solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below. In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0166] Figure 16 shows a schematic diagram of an image processing apparatus provided in an exemplary embodiment of this application; the image processing apparatus can be used to perform some or all of the steps in the method embodiments shown in Figures 7, 11, or 15. Referring to Figure 16, the apparatus includes the following units:

[0167] The acquisition unit 1601 is used to acquire a reference image and a content description text. The reference image is used to generate a target video, and the target video includes a frame sequence consisting of N video frames. The content description text is used to describe the video content that the target video should contain. N is a positive integer.

[0168] The processing unit 1602 is used to perform encoding copying processing on the reference image to obtain N image encoding features to be attenuated. The image encoding features to be attenuated are used to characterize the content characteristics of the reference image content.

[0169] The processing unit 1602 is further configured to perform feature attenuation processing on N image coding features to be attenuated to obtain N attenuated image coding features; one attenuated image coding feature is used to generate one video frame; the feature attenuation processing is used to control the content difference between the image content of each video frame and the reference image content;

[0170] The processing unit 1602 is also used to perform video generation processing on N attenuated image coding features according to the content description text to generate a target video; the image content of each video frame in the generated target video matches the video content and has content differences with the reference image content.

[0171] In one implementation, the processing unit, when performing feature attenuation processing on N image coding features to be attenuated to obtain N attenuated image coding features, specifically performs the following:

[0172] Obtain the attenuation parameter, which is used to characterize the degree of content difference between adjacent video frames.

[0173] N attenuation coefficients are calculated using attenuation parameters; each of the N attenuation coefficients corresponds one-to-one with the N encoded features of the image to be attenuated.

[0174] Each attenuation coefficient is multiplied by its corresponding image coding feature to be attenuated to obtain the attenuated image coding feature corresponding to each image feature to be attenuated.

[0175] In one implementation, the processing unit, used to perform coded copying of the reference image to obtain N coded features of the image to be attenuated, specifically performs the following:

[0176] Image coding processing is performed on the reference image to obtain the coded features of the image to be attenuated;

[0177] The image coding features to be attenuated are copied N times to obtain N image coding features to be attenuated.

[0178] In one implementation, the image coding features to be attenuated include latent space reference features to be attenuated. A processing unit, used to perform image coding processing on the reference image to obtain the image coding features to be attenuated, specifically performs the following:

[0179] The reference image is mapped to the latent space to obtain the reference features of the latent space to be attenuated.

[0180] In one implementation, the image encoding features to be attenuated include an image feature vector to be attenuated. A processing unit, used to perform image encoding processing on a reference image to obtain the image encoding features to be attenuated, specifically performs the following:

[0181] Image preprocessing is performed on the reference image to obtain an initial image feature vector; the initial image feature vector contains the feature vector of the image to be attenuated and preprocessing information, and the preprocessing information is used to indicate the image analysis information of the reference image;

[0182] Extract the image feature vector to be attenuated from the initial image feature vector.

[0183] In one implementation, the processing unit, when extracting the image feature vector to be attenuated from the initial image feature vector, specifically performs the following:

[0184] Obtain the initial image semantic vector, which indicates the weight distribution used when performing content extraction processing from the initial image feature vector;

[0185] The initial image semantic vector and the initial image feature vector are input into the cross-attention layer for attention operation, and the output of the cross-attention layer is input into the fully connected layer for fully connected operation to obtain the image feature vector to be attenuated.

[0186] In one implementation, content extraction processing is performed by content extraction modules, with M modules representing one content extraction process. M is a positive integer. The input information for each content extraction module includes:

[0187] When M=1, the input information of the first content extraction module is: the initial image semantic vector and the initial image feature vector;

[0188] When M=2, the input information of the second content extraction module is: the concatenated information obtained by concatenating the output information of the first content extraction module and the initial image semantic vector;

[0189] When M > 2, the input information of the Mth content extraction module is the concatenated information obtained by concatenating the output information of the (M-1)th content extraction module and the output information of the (M-2)th content extraction module.

[0190] In one implementation, the processing unit is used to perform video generation processing on N attenuated image coding features according to the content description text, specifically for generating the target video:

[0191] N latent space noise initialization features are obtained, which are obtained by adding random noise to blank latent space reference features; one latent space noise initialization feature is used to generate one video frame; and,

[0192] Text recognition processing is performed on the content description text to obtain a text recognition vector; the text recognition vector is used to represent the semantic information of the content description text.

[0193] The video generation model is invoked to process the text recognition vector and N latent space noise initialization features for video generation. During the video generation process, N attenuated image coding features are embedded to obtain the target video.

[0194] In one implementation, the attenuated image coding features include attenuated latent space reference features. A processing unit is used to invoke a video generation model to perform video generation processing on the text recognition vector and N latent space noise initialization features, and to embed N attenuated image coding features during the video generation process. Specifically, when the target video is obtained, it is used for:

[0195] N attenuated latent space reference features are embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set; the latent space reference feature set includes N latent space target features, and each latent space target feature is obtained by embedding an attenuated latent space reference feature into the corresponding latent space noise initialization feature.

[0196] The video generation model is invoked to process the latent space reference feature set and text recognition vectors to generate the target video.

[0197] In one implementation, the video generation model includes a multi-level feature extraction module; the attenuated image coding features include the attenuated image feature vector;

[0198] The processing unit, used to embed N attenuated image coding features during video generation and processing, is specifically used for:

[0199] During the video generation process, N attenuated image feature vectors are embedded into the feature extraction module at each level.

[0200] In one implementation, the feature extraction module includes a first attention submodule corresponding to the text and a second attention submodule corresponding to the image; any level of feature extraction module is represented as a target feature extraction module; the processing unit, used to embed N attenuated image feature vectors into each level of feature extraction module, specifically performs the following:

[0201] The second attention submodule within the target feature extraction module performs attention operations on N attenuated image feature vectors to obtain the second attention operation result; and...

[0202] The first attention submodule in the target feature extraction module performs attention operations on the text recognition vector to obtain the first attention operation result.

[0203] The output information of the target feature extraction module is obtained by fusing the results of the second attention operation and the first attention operation.

[0204] In one implementation, the video generation model includes multi-level feature extraction modules, each level of which embeds a spatiotemporal attention module; the processing unit is further used for:

[0205] The spatiotemporal attention module embedded in the feature extraction module is used to perform prediction processing on the output information of the feature extraction module to obtain the prediction result. The prediction result is used to indicate the content correlation between the image content of two adjacent video frames when generating a frame sequence based on the output information of the feature extraction module.

[0206] According to one embodiment of this application, the various units in the image processing apparatus shown in FIG16 can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This can achieve the same operation without affecting the technical effect of the embodiment of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units. According to another embodiment of this application, the image processing apparatus shown in FIG16 and the image processing method of the embodiments of this application can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding methods shown in FIG7, FIG11, or FIG15 on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0207] In this embodiment, on the one hand, since the N attenuated image coding features are obtained based on the N image coding features to be attenuated, and the N image coding features to be attenuated indicate the content characteristics of the image content of the reference image, when the video generation model is called to perform video generation processing on the N attenuated image coding features to obtain the target video, it relies on the N attenuated image coding features to characterize the content characteristics of the image content of the reference image to different degrees, so that the image content of each video frame in the frame sequence of the target video generated based on the N attenuated image coding features has content consistency with the image content of the reference image, such as the subject in the image (such as people, environment, etc.) is the same subject, while having content differences with the image content of the reference image. On the other hand, during the process of calling the generative model to process N attenuated image coding features to obtain the target video, the reference to the reference image can be reduced frame by frame during the process of calling the video generation model to generate N video frames based on the reference image. This helps the video generation model introduce more image content described by the content description text to constrain / guide the generation of the target video. This not only ensures that the image content of each video frame in the frame sequence included in the target video frame matches the image content described by the content description text (such as matching the appearance of the subject in the video frame (e.g., matching a person's actions with the actions described in the content description text), but also that the image content described by the content description text can assist in the generation of image elements in the content difference parts between the image content of the video frame and the image content of the reference image. Furthermore, each of the N attenuated image coding features is used to generate one video frame. Since the N attenuated image coding features are not the same, this ensures that the image content of the N video frames is also different, thereby ensuring that the target video can present the content change effect.

[0208] Figure 17 shows a schematic diagram of a computer device provided in an exemplary embodiment of this application. Referring to Figure 17, the computer device includes a processor 1701, a communication interface 1702, and a computer-readable storage medium 1703. The processor 1701, communication interface 1702, and computer-readable storage medium 1703 can be connected via a bus or other means. The communication interface 1702 is used to receive and send data. The computer-readable storage medium 1703 can be stored in the memory of the computer device and is used to store computer programs. The processor 1701 is used to execute the computer programs stored in the computer-readable storage medium 1703. The processor 1701 (or CPU) is the computing and control core of the computer device, and it is suitable for implementing one or more computer programs, specifically for loading and executing one or more computer programs to achieve corresponding method flows or corresponding functions.

[0209] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores one or more computer programs suitable for loading and execution by the processor 1701. These computer programs can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0210] In one embodiment, the computer device may be the terminal or server mentioned in the foregoing embodiments; the computer-readable storage medium stores one or more computer programs; the processor 1701 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-described image processing method embodiments; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded and executed by the processor 1701 to execute the steps of the embodiments of this application; wherein, the steps of the embodiments of this application can be referred to the relevant descriptions of the foregoing embodiments, and will not be repeated here.

[0211] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0212] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image processing method.

[0213] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0214] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer programs. When a computer program is loaded and executed on a computer, it generates, in whole or in part, the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in or transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital video disks, DVDs), or semiconductor media (e.g., solid state disks, SSDs), etc.

[0215] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is performed by a computer device, and the method includes: A reference image and content description text are obtained. The reference image is used to generate a target video, which includes a frame sequence consisting of N video frames. The content description text is used to describe the video content that the target video should contain. N is a positive integer. The reference image is encoded and copied to obtain N image encoding features to be attenuated. These image encoding features are used to characterize the content characteristics of the reference image content. The N image coding features to be attenuated are subjected to feature attenuation processing to obtain N attenuated image coding features; one attenuated image coding feature is used to generate one video frame; the feature attenuation processing is used to control the content difference between the image content of each video frame and the content of the reference image; According to the content description text, video generation processing is performed on N attenuated image coding features to generate the target video; the image content of each video frame in the generated target video matches the video content and has content differences from the reference image content.

2. The method as described in claim 1, characterized in that, The step of performing feature attenuation processing on the N image coding features to be attenuated to obtain N attenuated image coding features includes: Obtain the attenuation parameter, which is used to characterize the degree of content difference between the frame image content of adjacent video frames; N attenuation coefficients are calculated using the attenuation parameters; each of the N attenuation coefficients corresponds one-to-one with the N encoded features of the image to be attenuated. Each attenuation coefficient is multiplied by its corresponding image coding feature to be attenuated to obtain the attenuated image coding feature corresponding to each image feature to be attenuated.

3. The method as described in claim 1 or 2, characterized in that, The process of encoding and copying the reference image to obtain N image encoding features to be attenuated includes: The reference image is subjected to image encoding processing to obtain the image encoding features to be attenuated; The image coding feature to be attenuated is copied N times to obtain N image coding features to be attenuated.

4. The method according to any one of claims 1-3, characterized in that, The image coding features to be attenuated include latent space reference features to be attenuated. The image coding process performed on the reference image to obtain the image coding features to be attenuated includes: The reference image is mapped to the latent space to obtain the latent space reference features to be attenuated.

5. The method according to any one of claims 1-4, characterized in that, The image coding features to be attenuated include an image feature vector to be attenuated. The image coding process performed on the reference image to obtain the image coding features to be attenuated includes: The reference image is preprocessed to obtain an initial image feature vector; the initial image feature vector contains the image feature vector to be attenuated and preprocessing information, and the preprocessing information is used to indicate the image analysis information of the reference image. The image feature vector to be attenuated is extracted from the initial image feature vector.

6. The method according to any one of claims 1-5, characterized in that, The step of extracting the image feature vector to be attenuated from the initial image feature vector includes: Obtain an initial image semantic vector, which is used to indicate the weight distribution used when performing content extraction processing from the initial image feature vector; The initial image semantic vector and the initial image feature vector are input into the cross-attention layer for attention operation, and the output of the cross-attention layer is input into the fully connected layer for fully connected operation to obtain the image feature vector to be attenuated.

7. The method according to any one of claims 1-6, characterized in that, The content extraction process is performed by a content extraction module. The number of content extraction modules is M, and one content extraction module corresponds to one content extraction process. M is a positive integer. The input information for the content extraction module includes: When M=1, the input information of the first content extraction module is: the initial image semantic vector and the initial image feature vector; When M=2, the input information of the second content extraction module is: the concatenated information obtained by concatenating the output information of the first content extraction module and the initial image semantic vector; When M > 2, the input information of the Mth content extraction module is the concatenated information obtained by concatenating the output information of the (M-1)th content extraction module and the output information of the (M-2)th content extraction module.

8. The method according to any one of claims 1-7, characterized in that, The step of generating the target video by performing video generation processing on N attenuated image coding features according to the content description text includes: Obtain N latent space noise initialization features, wherein the latent space noise initialization features are obtained by adding random noise to blank latent space reference features; one of the latent space noise initialization features is used to generate one video frame; and, The content description text is subjected to text recognition processing to obtain a text recognition vector; the text recognition vector is used to represent the semantic information of the content description text. The video generation model is invoked to perform video generation processing on the text recognition vector and N latent space noise initialization features, and N attenuated image coding features are embedded in the video generation process to obtain the target video.

9. The method according to any one of claims 1-8, characterized in that, The attenuated image coding features include attenuated latent space reference features. The process of calling the video generation model to perform video generation processing on the text recognition vector and N of the latent space noise initialization features, and embedding the N attenuated image coding features during the video generation process, yields the target video, including: N attenuated latent space reference features are embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set; the latent space reference feature set includes N latent space target features, and each latent space target feature is obtained by embedding one attenuated latent space reference feature into the corresponding latent space noise initialization feature. The target video is obtained by calling a video generation model to process the latent space reference feature set and the text recognition vector.

10. The method according to any one of claims 1-9, characterized in that, The video generation model includes multi-level feature extraction modules; The attenuated image coding features include the attenuated image feature vector; The embedding of N attenuated image coding features during the video generation process includes: During the video generation process, N attenuated image feature vectors are embedded into the feature extraction module at each level.

11. The method according to any one of claims 1-10, characterized in that, The feature extraction module includes a first attention submodule corresponding to text and a second attention submodule corresponding to images; the feature extraction module at any level is referred to as the target feature extraction module; The step of embedding the N attenuated image feature vectors into the feature extraction module at each level includes: The second attention submodule in the target feature extraction module performs attention operations on the N attenuated image feature vectors to obtain the second attention operation result; and... The first attention submodule in the target feature extraction module performs attention operations on the text recognition vector to obtain the first attention operation result. The output information of the target feature extraction module is obtained by fusing the second attention operation result and the first attention operation result.

12. The method according to any one of claims 1-11, characterized in that, The video generation model includes multi-level feature extraction modules, and each level of the feature extraction module embeds a spatiotemporal attention module; the method further includes: The spatiotemporal attention module embedded in the feature extraction module is used to perform prediction processing on the output information output by the feature extraction module to obtain a prediction result; the prediction result is used to indicate the content correlation between the image content of two adjacent video frames when the frame sequence is generated based on the output information output by the feature extraction module.

13. An image processing apparatus, characterized in that, The image processing device is mounted on a computer device, and the image processing device includes: An acquisition unit is used to acquire a reference image and content description text. The reference image is used to generate a target video, and the target video includes a frame sequence consisting of N video frames. The content description text is used to describe the video content that the target video should contain. N is a positive integer. The processing unit is used to perform encoding and copying processing on the reference image to obtain N image encoding features to be attenuated, wherein the image encoding features to be attenuated are used to characterize the content characteristics of the reference image content of the reference image; The processing unit is further configured to perform feature attenuation processing on the N image coding features to be attenuated to obtain N attenuated image coding features; one attenuated image coding feature is used to generate one video frame; the feature attenuation processing is used to control the content difference between the image content of each video frame and the reference image content; The processing unit is further configured to perform video generation processing on the N attenuated image coding features according to the content description text to generate the target video; the image content of each video frame in the generated target video matches the video content and has content differences from the reference image content.

14. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program that, when executed by the processor, implements the image processing method as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the image processing method as described in any one of claims 1-12.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image processing method as described in any one of claims 1-12.

Images