Image processing method, device, equipment, medium and program product

Abstract

The embodiments of the present application disclose an image processing method, device, equipment, medium and program product. The method includes: obtaining a reference image and a content description text; performing encoding and copying processing on the reference image to obtain N coding features of the image to be attenuated; performing feature attenuation processing on the N coding features of the image to be attenuated to obtain N coding features of the attenuated image; performing video generation processing on the N coding features of the attenuated image according to the image content described by the content description text to obtain a target video. The use of the embodiments of the present application can ensure that the image content of each video frame in the target video has both content difference and content consistency with the image content of the reference image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of artificial intelligence, and specifically to an image processing method, device, equipment, medium and program product. Background Art

[0002] Image-to-video conversion refers to the process of generating a video based on a single image that matches the image content of the single image.

[0003] Currently, it is supported to use diffusion models to generate videos based on images. For example, when adding a time series dimension to the diffusion model, it will cause too much loss of content information in the reference image, resulting in the image content of each video frame in the video being inconsistent with the image content of the reference image. For another example, if only the encoding features for the reference image are added to the production stage of the latent space features (latent), there will be a problem that the denoising generation stage cannot be consistent with the image content of the reference image. In addition, both of the above-mentioned two related technologies have the risk of the image content of each video frame in the generated frame sequence being exactly the same, resulting in the generated video content not having a content change effect.

[0004] Therefore, how to realize image to video conversion has become a research hotspot. Summary of the invention

[0005] The embodiments of the present application provide an image processing method, apparatus, device, medium and program product, which can ensure that the image content of each video frame in the target video matches the image content of each video frame in the target video described by the content description text during the process of converting a reference image to a target video, while at the same time ensuring that the image content has content difference and content consistency with the image content of the reference image.

[0006] On the one hand, an embodiment of the present application provides an image processing method, the method comprising:

[0007] A reference image and a content description text are obtained; the reference image is used to generate a target video, and the target video includes a frame sequence consisting of N video frames; the content description text is used to describe the image content that each video frame in the frame sequence should contain; N is a positive integer;

[0008] Performing encoding and duplication processing on the reference image to obtain N encoding features of the image to be attenuated, where the encoding features of the image to be attenuated are used to characterize content characteristics of the image content of the reference image;

[0009] Performing feature attenuation processing on N to-be-attenuated image coding features to obtain N attenuated image coding features; each attenuated image coding feature is used to generate a video frame in a frame sequence; the feature attenuation processing is used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image;

[0010] According to the image content described by the content description text, video generation processing is performed on N attenuated image coding features to obtain a target video; the image content of each video frame in the frame sequence included in the target video matches the image content described by the content description text, and has content difference with the image content of the reference image.

[0011] On the other hand, an embodiment of the present application provides an image processing device, the device comprising:

[0012] The acquisition unit is used to acquire a reference image and a content description text; the reference image is used to generate a target video, and the target video includes a frame sequence composed of N video frames; the content description text is used to describe the image content that each video frame in the frame sequence should contain; N is a positive integer;

[0013] A processing unit, used for performing coding and copying processing on the reference image to obtain N coding features of the image to be attenuated, wherein the coding features of the image to be attenuated are used to characterize content characteristics of the image content of the reference image;

[0014] The processing unit is further used to perform feature attenuation processing on the N to-be-attenuated image coding features to obtain N attenuated image coding features; each attenuated image coding feature is used to generate a video frame in the frame sequence; the feature attenuation processing is used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image;

[0015] The processing unit is further used to perform video generation processing on N attenuated image coding features according to the image content described by the content description text to obtain a target video; the image content of each video frame in the frame sequence included in the target video matches the image content described by the content description text, and has content differences with the image content of the reference image.

[0016] In one implementation, the processing unit is used to perform feature attenuation processing on N to-be-attenuated image coding features to obtain N attenuated image coding features, specifically for:

[0017] Obtaining a preset attenuation parameter, where the attenuation parameter is used to indicate a degree of content difference between image contents of adjacent video frames in a frame sequence;

[0018] The attenuation parameters are used to calculate N attenuation coefficients; the N attenuation coefficients correspond one to one to the N encoding features of the image to be attenuated;

[0019] The N attenuation coefficients are multiplied by the corresponding coding features of the image to be attenuated to obtain N coding features of the attenuated image.

[0020] In one implementation, the processing unit is used to perform coding and copying processing on the reference image to obtain N coding features of the image to be attenuated, specifically for:

[0021] Performing image coding processing on the reference image to obtain coding features of the image to be attenuated;

[0022] The coding feature of the image to be attenuated is copied N times to obtain N coding features of the image to be attenuated.

[0023] In one implementation, the coding feature of the image to be attenuated includes a reference feature of the latent space to be attenuated, and the processing unit is used to perform image coding processing on the reference image to obtain the coding feature of the image to be attenuated, specifically for:

[0024] The reference image is mapped to the latent space to obtain the reference features of the latent space to be attenuated.

[0025] In one implementation, the coding feature of the image to be attenuated includes a feature vector of the image to be attenuated, and the processing unit is used to perform image coding processing on the reference image to obtain the coding feature of the image to be attenuated, specifically for:

[0026] Performing image preprocessing on the reference image to obtain an initial image feature vector; the initial image feature vector includes a feature vector of the image to be attenuated and preprocessing information, and the preprocessing information indicates image analysis information for the reference image;

[0027] The initial image feature vector is subjected to content extraction processing to obtain the feature vector of the image to be attenuated.

[0028] In one implementation, the processing unit is used to perform content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated, specifically to:

[0029] Obtaining a preset initial image semantic vector, where the initial image semantic vector is used to indicate a weight distribution for content extraction processing from the initial image feature vector;

[0030] Based on the initial image semantic vector, the initial image feature vector is subjected to content extraction processing to obtain the feature vector of the image to be attenuated; the content extraction processing includes an attention operation and a full connection operation.

[0031] In one implementation, the content extraction process is performed by a content extraction module. The number of content extraction modules is M. One content extraction module corresponds to one content extraction process. M is a positive integer. The input information of the content extraction module includes:

[0032] When M=1, the input information of the first content extraction module is: initial image semantic vector and initial image feature vector;

[0033] When M=2, the input information of the second content extraction module is: the output information of the first content extraction module and the concatenation information of the initial image semantic vector;

[0034] When M>2, the input information of the Mth content extraction module is: the concatenated information of the output information of the M-1th content extraction module and the output information of the M-2th content extraction module.

[0035] In one implementation, the processing unit is used to perform video generation processing on N attenuated image coding features according to the image content described by the content description text to obtain the target video, specifically for:

[0036] Obtaining N latent space noise initialization features, where the latent space noise initialization features are obtained by adding random noise to a blank latent space reference feature; one latent space noise initialization feature is used to generate a video frame in a frame sequence; and,

[0037] Perform text recognition processing on the content description text to obtain a text recognition vector; the text recognition vector is used to represent the semantic information of the content description text;

[0038] The video generation model is called to perform video generation processing on the text recognition vector and N latent space noise initialization features, and N attenuated image coding features are embedded in the video generation process to obtain the target video.

[0039] In one implementation, the attenuated image coding features include attenuated latent space reference features, and the processing unit is used to call the video generation model to perform video generation processing on the text recognition vector and the N latent space noise initialization features, and embed the N attenuated image coding features in the process of video generation processing to obtain the target video, specifically for:

[0040] The N attenuated latent space reference features are respectively embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set; the latent space reference feature set includes N latent space target features, and a latent space target feature is obtained by embedding an attenuated latent space reference feature into a corresponding latent space noise initialization feature;

[0041] The video generation model is called to perform video generation processing on the latent space reference feature set and the text recognition vector to obtain the target video.

[0042] In one implementation, the video generation model includes a multi-level feature extraction module;

[0043] The attenuated image coding features include an attenuated image feature vector, and a processing unit, which is used to embed N attenuated image coding features during the video generation process, specifically for:

[0044] During the video generation process, N attenuated image feature vectors are embedded into each level feature extraction module in the video generation model.

[0045] In one implementation, the feature extraction module includes a first attention submodule corresponding to the text and a second attention submodule corresponding to the image; any level feature extraction module in the multi-level feature extraction module is represented as a target feature extraction module; and the processing unit is used to embed N attenuated image feature vectors into each level feature extraction module in the video generation model, specifically for:

[0046] Performing an attention operation on the N attenuated image feature vectors through a second attention submodule in the target feature extraction module to obtain a second attention operation result; and

[0047] Performing an attention operation on the text recognition vector through the first attention submodule in the target feature extraction module to obtain a first attention operation result;

[0048] The second attention operation result and the first attention operation result are fused to obtain the output information of the target feature extraction module.

[0049] In one implementation, the video generation model includes multiple layers of feature extraction modules, each layer of feature extraction modules is embedded with a spatiotemporal attention module; the processing unit is further used to:

[0050] The spatiotemporal attention module is used to predict the output information output by the corresponding feature extraction module to obtain a prediction result; the prediction result is used to indicate: when a frame sequence is generated based on the output information output by the corresponding feature extraction module, the content correlation between the image contents of two adjacent video frames.

[0051] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising:

[0052] a processor adapted to execute a computer program;

[0053] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image processing method is implemented.

[0054] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned image processing method.

[0055] On the other hand, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0056] In an embodiment of the present application, it supports obtaining a reference image as the generation guide information of the video content of the target video, and obtaining a content description text to describe the image content of each video frame in the frame sequence included in the target video. Then, the reference image is copied and encoded to obtain N image coding features to be attenuated; any image coding feature to be attenuated is used to characterize the content characteristics of the image content of the reference image, and the N image coding features to be attenuated correspond to the number N of video frames in the frame sequence included in the target video to be generated. In order to ensure that the image content of each video frame in the frame sequence included in the target video has content differences, that is, to ensure that the picture content presented by the target video is changing, it is necessary to perform feature attenuation processing on the N image coding features to be attenuated to obtain N attenuated image coding features; each of the N attenuated image coding features has feature differences with the image coding feature to be attenuated. On the one hand, since the N attenuated image coding features are obtained based on the N to-be-attenuated image coding features, and the N to-be-attenuated image coding features indicate the content characteristics of the image content of the reference image, when the N attenuated image coding features are subjected to video generation processing to obtain the target video, the N attenuated image coding features are relied upon to characterize the content characteristics of the image content of the reference image from different feature degrees, so that the image content of each video frame in the frame sequence included in the target video generated based on the N attenuated image coding features has content consistency with the image content of the reference image, such as the subject in the image (such as a person, environment, etc.) appears as the same subject, and at the same time, there is content difference between the image content and the reference image. On the other hand, in the process of performing video generation processing on N attenuated image coding features to obtain the target video, the reference to the reference image can be reduced frame by frame in the process of generating N video frames according to the reference image, so that more image content described by the content description text is introduced in the video generation process to constrain / guide the generation of the target video; not only can it be ensured that the image content of each video frame in the frame sequence included in the target video frame matches the image content described by the content description text, such as the expression form of the subject in the video frame matches (such as the action of a person matches the action described by the content description text), but also the image content described by the content description text can assist in the generation of image elements of the content difference part between the image content of the video frame and the image content of the reference image. On the other hand, each of the N attenuated image coding features is used to generate a video frame. Since the N attenuated image coding features are not the same, it can be ensured that the image content of the N video frames is also different, thereby ensuring that the target video can present the effect of content change. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1a It is a schematic diagram of a basic training framework of a Stable Diffusion model provided by an exemplary embodiment of the present application;

[0059] Figure 1b It is a schematic diagram of a reasoning basic framework of a Stable Diffusion model provided by an exemplary embodiment of the present application;

[0060] Figure 2 It is a schematic diagram of a model structure of a video generation model obtained by embedding a spatiotemporal attention module in a Stable Diffusion model provided by an exemplary embodiment of the present application;

[0061] Figure 3 It is a schematic diagram of the architecture of an image processing system in a case where a feature embedding mechanism provided by an exemplary embodiment of the present application is a latent space embedding mechanism;

[0062] Figure 4a is a schematic diagram of a model structure after fine-tuning a feature extraction module in a video generation model provided by an exemplary embodiment of the present application;

[0063] Figure 4b is a schematic diagram of the architecture of an image processing system in a case where a feature embedding mechanism provided by an exemplary embodiment of the present application is a multi-level cross-attention mechanism;

[0064] Figure 5 It is a schematic diagram of the architecture of an image processing system in which a feature embedding mechanism is a dual-stream consistency embedding mechanism provided by an exemplary embodiment of the present application;

[0065] Figure 6 It is a schematic diagram of a scenario architecture of an application scenario provided by an exemplary embodiment of the present application;

[0066] Figure 7 is a flowchart of an image processing method provided by an exemplary embodiment of the present application;

[0067] Figure 8 It is a schematic diagram of an interface for uploading a reference image and content description text provided by an exemplary embodiment of the present application;

[0068] Fig. 9 is a schematic diagram of a calculation flow of a feature attenuation process provided by an exemplary embodiment of the present application;

[0069] Fig.10 It is a schematic diagram of embedding N attenuated latent space reference features into N latent space noise initialization features provided by an exemplary embodiment of the present application;

[0070] Fig.11 is a flowchart of another image processing method provided by an exemplary embodiment of the present application;

[0071] Fig.12 It is a schematic diagram of a flow chart of an image encoder used in a multi-level cross-attention embedding mechanism, which performs image encoding processing on a reference image, provided by an exemplary embodiment of the present application;

[0072] Fig.13 It is another flowchart of image encoding processing for a reference image in the case of a multi-level cross attention embedding mechanism provided by an exemplary embodiment of the present application;

[0073] Fig.14 is a schematic diagram of a structure of a fine-tuned feature extraction module provided by an exemplary embodiment of the present application;

[0074] Fig.15 is a flowchart of another image processing method provided by an exemplary embodiment of the present application;

[0075] Fig.16 is a structural schematic diagram of an image processing device provided by an exemplary embodiment of the present application;

[0076] Fig.17 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] In an embodiment of the present application, an image processing scheme is proposed, specifically, an image-to-video generation scheme. Image-to-video refers to the process of converting a static image into a dynamic video; that is, a static image is provided as a content reference so that the corresponding video can be converted with reference to the static image. The converted video includes a frame sequence consisting of N video frames, where N is a positive integer; thus, the process of converting the image is essentially to generate N video frames based on the image, and the N video frames form a video (i.e., a video frame sequence, or simply a frame sequence) in the order of playback time.

[0079] In the image-to-video technology, if it can be ensured that the image content of each video frame in the frame sequence has content consistency and content difference with the image content of the static image, then it is determined that the image-to-video has good content conversion capabilities and can convert high-quality videos. Among them, ① the image content of each video frame in the frame sequence obtained by image-to-video conversion needs to maintain content consistency with the image content of the static image. The so-called content consistency can be understood as: the subject (or subject content) in the image content of each video frame is the same as the subject in the image content of the static image. The subject in the image content (or the subject in the image or video frame) can be understood as the main object in the image / video frame, that is, the object that is to be highlighted in the image / video frame. For example, in a person image, the object that is to be highlighted in the person image is the person itself; then, after generating a video based on the person image, it is necessary to ensure that the person itself included in each video frame in the video and the person itself in the person image are the same, such as the appearance and clothing of the person are the same, emphasizing that the person itself in the video frame and the person itself in the person image can be recognized as the same person. In a landscape image, the objects that one wants to highlight in the landscape image are mountains and lakes, etc. Therefore, after generating a video based on the landscape image, it is necessary to ensure that the mountains and lakes included in each frame of the video are the same as the mountains and lakes in the landscape image, such as the relative positions of the mountains and lakes are the same, the shapes of the mountains are the same, the shapes of the lakes are the same, and so on.

[0080] ② The image content of each video frame in the frame sequence obtained by image-to-video conversion needs to maintain content differences with the image content of the static image. The so-called content difference can be understood as: the expression form of the subject in the image content of each video frame is different from the expression form of the subject in the image content of the static image. For example, when the static image is a person image, the expression form of the person itself may include but is not limited to: the expression form of expression (such as smiling, pouting, blinking and pursing lips, etc.), the expression form of limbs (such as walking, stillness, jumping and running, etc.). For another example, when the static image is a landscape image, and the subject in the landscape image is a mountain and a lake, the expression form of the mountain may include objects on the mountain (such as trees and grass, etc.) floating in the wind, and the expression form of the lake may include the form of water waves flowing on the lake surface, etc. By controlling the content difference between the image content of the static image and the image content of each video frame, it is ensured that the video presents a dynamic visual effect, thereby meeting the image-to-video conversion requirements of generating dynamic videos based on static images.

[0081] To ensure that the video generated by the image-to-video technology can meet the above-mentioned content consistency and content difference; that is, to ensure that the video content of the video generated by the image-to-video technology is generated based on the desired main content (i.e., the main body in the image content of the static image), and can present the dynamic effect of the desired performance form. The image-to-video generation scheme provided in the embodiment of the present application is specifically a generation scheme of image-to-video based on attenuation feature injection, and the general principles of the scheme can include:

[0082] A reference image and a content description text are obtained; the reference image is used to generate a target video (the target video includes a frame sequence consisting of N video frames), specifically, the main content in the image content of the reference image is used to generate the main content in the image content of each video frame in the frame sequence, so as to ensure that the image content of each video frame in the frame sequence has content consistency with the image content of the reference image; the content description text is used to describe the image content that each video frame in the frame sequence should contain, and is specifically used to guide the expression morphological change trend of the subject in the image content of each video frame in the frame sequence, such as at least the action change of the character in the image content of the video frame. Then, the reference image is coded and copied to obtain N image coding features to be attenuated, and the image coding features to be attenuated are used to identify the content characteristics of the image content of the reference image; in other words, the coding and copying process performed on the reference image is intended to extract the main features in the image content of the reference image, and copy the features into N copies, each corresponding to a video frame. Then, N attenuated image coding features are subjected to feature attenuation processing to obtain N attenuated image coding features; the feature attenuation processing can be used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image, that is, to ensure that when the attenuated image coding is used to generate a video frame, the image content of the video frame can maintain a certain content difference with the image content of the reference image. In this way, according to the image content of each video frame described by the content description text, after the N attenuated image coding features are subjected to video generation processing, a target video having both content difference and content consistency between the image content of each video frame and the image content of the reference image can be obtained. Among them, the role of the content description text in the video generation process is to introduce the image content described by the content description text when generating a video frame with content difference from the image content of the reference image, so as to guide the difference change point or trend of the content difference; for example, the difference change point between the expression form of the face in the image content of the reference image (such as the lip bending amplitude) and the expression form of the face in the image content of the video frame is guided by the content description text.

[0083] It can be seen that, on the one hand, the embodiment of the present application introduces a reference image (represented as ref image) as a base image for generating the target video; the base image is used as reference information in the entire video generation process, that is, the content information in the base image is referenced in the entire frame sequence generation process, ensuring that the image content of each video frame in the generated frame sequence and the image content in the base image have content consistency, preventing sudden changes in the content of the target video, and greatly improving the content retention of the overall content of the target video to the image content of the reference image. On the other hand, the image coding features to be attenuated extracted from the reference image need to be processed by feature attenuation before being embedded in the video generation process; this attenuated embedding method can realize the attenuation of the guidance information (i.e., the image coding features to be attenuated) calculated based on the ref image following time (such as the playback time of the frame sequence), thereby guiding the generation of image content in video frames corresponding to different timestamps according to the attenuated image coding features, ensuring that the image content of each video frame in the frame sequence and the image content of the reference image have content differences, and the difference points of the content differences are guided by the content description text, thereby improving the video quality of the target video, specifically ensuring that the overall video content of the target video meets user requirements.

[0084] The image processing solution proposed in the embodiment of the present application is implemented based on the video generation model; specifically, after obtaining N attenuated image coding features of the reference image, it is necessary to embed the N attenuated image coding features into the process of calling the video generation model for video generation processing, so as to realize the conversion of the reference image into the target video. Among them, the video generation model involved in the embodiment of the present application is obtained by improving the basic diffusion model; the improvement lies in: adding a spatiotemporal attention module to the basic diffusion model, and adding an attention mechanism to the basic diffusion model. In order to better understand the architecture and principles of the video generation model, the basic diffusion model and the spatiotemporal attention module are first introduced below, among which:

[0085] (1) Diffusion model

[0086] The diffusion model (or simply the Wenshengtu diffusion model) is a pre-trained model that can be viewed as a series of denoising autoencoders; its purpose is to eliminate the continuous application of Gaussian noise to the training image, and the input text controls the generation of pictures (or images) during the pre-training diffusion model. In detail, the diffusion model includes two main processes: forward diffusion and reverse diffusion; in the forward diffusion stage, the image is contaminated by the gradually introduced noise until the image becomes completely random noise; in the reverse process, a series of Markov chains are used to gradually remove the prediction noise at each time step (or timestamp, such as 1 second), thereby recovering the data from the Gaussian noise to generate an image. Furthermore, there are many types of diffusion models. The embodiment of the present application takes the diffusion model as a logical data model (Latent Diffusion Models, LDM) as an example to introduce. The logical data model can be called a Stable Diffusion model, which diffuses in the latent space instead of the pixel space, can save memory, and combines the text semantic feedback from the Transformer to produce diverse and highly detailed images while retaining the semantic structure of the data.

[0087] The basic training framework of the Stable Diffusion model can be found in Figure 1a The inference framework of the Stable Diffusion model can be found in Figure 1b See also Figure 1a and Figure 1b It can be seen that the training basic framework and reasoning basic framework of the Stable Diffusion model include the following three parts: variational autoencoder, U-Net network and text encoder. Among them:

[0088] ① The variational autoencoder consists of two parts, namely the encoder part and the decoder part. Among them, the encoder part of the variational autoencoder can be called vae encoder, and the decoder part of the variational autoencoder can be called vaedecoder.

[0089] The encoder vae encoder can convert the reference image into a latent representation in a low-dimensional latent space (the use of variational autoencoders in traditional stable diffusion models: the encoder vae encoder is only used to encode sample images in the model training phase of the stable diffusion model, and the encoder vae encoder is not used in the model inference phase). The latent representation is used as the input data of the U-Net network. The decoder vae decoder performs the opposite operation to the encoder vae encoder. The decoder vae decoder converts the latent representation in the latent space back to a high-dimensional image (i.e., video frame). Among them, the latent representation refers to a low-dimensional vector learned from the reference image, which is a manifestation of the intrinsic and implicit features in the reference image. In the stable diffusion model, the latent representation usually refers to the latent variable, that is, the variable that is not directly observed in the stable diffusion model, which can be understood as the most basic feature in the entire model processing flow; in the stable diffusion model, the latent variable may represent unobserved factors, such as individual characteristics, behaviors, styles, and compositions, which may affect the diffusion process. The information space where the potential representation is located is called the latent space; the latent space is an abstract conceptual space, and the latent representation in the space represents a highly generalized or compressed form of the image features in the reference image; this generalization or compression is achieved through the encoder VAE encoder, that is, the encoder VAE encoder projects the reference image into the latent space, thereby obtaining the latent representation of the image features of the reference image, so as to mine more comprehensive, deep and implicit features in the reference image.

[0090] ②U-Net network.

[0091] U-Net network, or Unet network, is an algorithm for semantic segmentation using a fully convolutional network; the algorithm contains a symmetrical U-shaped structure of compression path and expansion path. The Unet network includes an encoder part (denoted as down sample) and a decoder part (denoted as up sample); the encoder part is used to compress the reference image into a low-resolution image, and the decoder part is used to decode the low-resolution image back to a high-resolution image. The Unet network also includes a jump connection, which is a structure located between the encoder part and the decoder part. It fuses features of different scales through jump connections, thereby improving the accuracy of segmentation. It can be seen that the Unet network includes three parts, namely the downsampling part, the jump connection part, and the upsampling part; each of these three parts has the same composition, but the direction of the feature size dimension is different. Each part includes a multi-level feature extraction module (denoted as Unet block), and each feature extraction module consists of a deep residual block (denoted as ResNet) and an attention block. The deep residual block can introduce residual connections to solve problems such as gradient explosion and gradient disappearance during model training. The attention block includes cross attention (expressed as crossattention), which is used to adjust the output of the content description text of the input model. The attention block also includes self-attention (expressed as self attention), which is used to make the quality of video frames generated by the Unet network more exquisite, and the self-attention is located between the deep residual block and the cross attention.

[0092] Among them, the jump connection part mentioned above can be used to prevent the Unet network from losing important information when the encoder is down-sampled, and the jump connection part is usually added between the down-sampled ResNet of the encoder and the up-sampled ResNet of the decoder. In addition, the cross-attention mechanism is usually used to process the association between multiple input sequences in order to capture important information interactions between different sequences in the model; in the Stable Diffusion model, cross-attention can help the model better understand the association between different parts, so as to more accurately predict the changes and impacts in the diffusion process. The self-attention mechanism is an important supplementary part in the Unet block of Stable Diffusion. Its input is the backbone feature (that is, the latent representation) of the Stable Diffusion model. The role of self attention is to enable all semantic representations in the backbone features to be further integrated, aiming to make the representation of the generated content clearer and enhance the Unet network's ability to reduce noise on the backbone features.

[0093] based on Figure 1a and Figure 1bIt can be seen that during the training process of the Stable Diffusion model, the reference image is encoded by the vaeen encoder and noise is added to generate a new latent representation; the latent representation latent is input into the Unet network for noise reduction and other operations; the latent signal output by the Unet network is input into the vae decoder for decoding to generate a frame sequence. During the reasoning process of the Stable Diffusion model, since there is no training image, its purpose is to generate a specific image content image based on the desired video content; therefore, the input data of the Stable Diffusion model is a pure noise latent representation latent, and there is no need to use the vae encoder for encoding. In this way, the pure noise latent representation latent is input into the Unet network for noise reduction and other processing, and then the vae decoder decodes the latent signal output by the Unet network to generate an image of the desired video content (in the basic diffusion model, the desired video content is provided by text).

[0094] ③Text encoder.

[0095] The text encoder converts the input concept text sequence into an embedding space that can be understood by the Unet network, which is the text recognition vector (expressed as text embedding) referred to in the general model. In the embodiment of the present application, the text encoder module in the clip model is supported as a text encoder; the text encoder module is a relatively simple encoder based on transformer (a deep learning model based on the self-attention mechanism). In order to calculate a good semantic representation of the concept text, it maps the tag sequence to the potential text embedding sequence, so as to obtain a better expected output. Among them, the clip model is an advanced deep learning model that can understand text and images; the clip model is specifically a multimodal model. By training the multimodal model, text and images can be matched, and the content in the image and the language describing the image can be learned. During the training process, the clip model learns to encode images and texts into a unified vector space, so that the clip model can understand the relationship between images and texts in language and vision; specifically, it learns to match images and texts through contrast learning, aiming to predict which image belongs to a given text or which text belongs to a given image by training the clip model. Therefore, the clip model can be used to identify elements such as objects, scenes, actions, etc. in the image, and it can also understand the text related to the image, such as tags, descriptions, titles, etc.

[0096] (2) Spatiotemporal Attention Module.

[0097] The spatiotemporal attention module is called the motion module, which is used to complete the construction of the video generation function provided by the Stable Diffusion model. Its main function is to calculate the correlation between frames in the frame sequence during the video generation and training process of the Stable Diffusion model, that is, to calculate the correlation between the image contents of two adjacent video frames, so as to smoothly generate the content gap between frames.

[0098] Based on the above introduction to the basic diffusion model Stable Diffusion and the spatiotemporal attention module motion module, the embodiment of the present application supports embedding the spatiotemporal attention module into the Stable Diffusion model, specifically, embedding the spatiotemporal attention module into the multi-level feature extraction module of the Unet network in the Stable Diffusion model. The model structure diagram of the video generation model obtained by embedding the spatiotemporal attention module in the Stable Diffusion model can be seen in Figure 2 ;like Figure 2 As shown, the method of adding the motion module to the Stable Diffusion model includes: embedding a motion module in each level of the feature extraction module in the multi-level feature extraction module of the Stable Diffusion model, that is, embedding a motion module after each Unet block in the Stable Diffusion model to form a video generation model. In this way, in the process of generating a video frame sequence at one time by the video generation model, after the feature extraction module Unet block performs denoising and other processing on the potential representation, the spatiotemporal attention module embedded in the feature extraction module can be used to predict the output information (i.e., the potential representation) output by the corresponding feature extraction module to obtain a prediction result; the prediction result is used to indicate: when generating a frame sequence based on the output information output by the corresponding feature extraction module, the content correlation between the image content of two adjacent video frames; if the content correlation is too large, it indicates that the content jumps between frames and is not smooth enough, then the content change can be appropriately slowed down during subsequent denoising and other processing to ensure the smoothness of the content between frames. It should be noted that Figure 2 The functions of each module in the model structure of the video generation model shown in FIG. Figure 1a and Figure 1b The relevant description is not repeated here.

[0099] Furthermore, considering that the embodiment of the present application introduces a reference image for guiding content generation in the model reasoning stage, in order to embed the features of the reference image into the model reasoning process, the embodiment of the present application will also embed the features of the reference image into the model reasoning process according to different feature embedding mechanisms for the reference image. Figure 2 The video generation model shown is fine-tuned to adaptively embed the features of the reference image into the model inference stage. In addition, according to the different feature embedding mechanisms for the reference image, the system architecture of the image-to-video generation solution based on the video generation model is also different. The feature embedding mechanism for the reference image, the fine-tuning of the video generation model to adapt to the feature embedding mechanism, and the corresponding system architecture are introduced below, where:

[0100] (1) The feature embedding mechanism is a latent space embedding mechanism.

[0101] The principle of the latent space embedding mechanism is: embedding the features of the reference image into the N latent space noise initialization features used when generating the frame sequence; each of the N latent space noise initialization features is obtained by adding random noise to the blank latent space reference feature, which can be understood as the process of adding random noise to a blank image.

[0102] Since the features of the reference image are embedded into the latent space noise initialization features used when generating the sequence, there is no need to change the internal structure of the video generation model. Therefore, when the feature embedding mechanism is the latent space embedding mechanism, the model structure of the corresponding video generation model is Figure 2 The model structure of the video generation model shown is the same.

[0103] When the feature embedding mechanism is a latent space embedding mechanism, the system architecture of the image processing system can be seen in Figure 3 .like Figure 3As shown, the image processing system at least includes a video generation model, input data of the video generation model (i.e., N latent space noise initialization features used when generating a frame sequence) and a first image processing module 301. The first image processing module 301 includes the encoder vae encoder and the feature attenuation submodule of the Stable Diffusion model. The encoder vae encoder performs image encoding on the reference image to map the reference image to the latent space, and obtains N latent space reference features to be attenuated in the latent space for representing the image features; then the feature attenuation submodule is used to perform feature attenuation processing on the N latent space reference features to be attenuated, and obtains N attenuated latent space reference features; then the N attenuated latent space reference features are embedded into the N latent space noise initialization features, and N latent space target features are obtained; in this way, after the N latent space target features are input into the video generation model, the video generation model can perform noise reduction and other processing based on the N latent space target features and the content description text (text), and then use the decoder vae decoder for decoding, so as to generate the target video based on the reference image and the content description text at one time.

[0104] It can be seen that this latent space embedding mechanism can directly pass the content information / features of the reference image to each latent space noise initialization feature used when generating the frame sequence, which is similar to the picture generation process of directly passing the content information / features of the reference image to N video frames, thereby effectively ensuring that the image content of each video frame in the final generated target video is consistent with the image content of the reference image. In addition, a feature attenuation submodule is deployed in the first image processing module, so that the feature attenuation submodule can perform feature attenuation processing on the N latent space reference features to be attenuated extracted based on the reference image, thereby ensuring that the image content of each video frame has content differences with the image content of the reference image; further, the specific differences in the content differences between the image contents of the video frames are described by the content description text of the input video generation model, so as to ensure that the overall video content of the generated target video meets the video generation requirements and improve the video quality of the target video.

[0105] (2) The feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0106] The principle of the multi-level cross-attention embedding mechanism is to embed the features of the reference image into the feature extraction module of each level in the Unet network in the video generation model, specifically into the cross-attention mechanism of the feature frame extraction module of each level.

[0107] Since the features of the reference image are embedded into the video generation model, it is necessary to fine-tune the module structure of the feature extraction module in the video generation model so that the fine-tuned feature extraction module can receive the features of the reference image and process the features. For example, the model structure diagram of the video generation model after fine-tuning the feature extraction module in the video generation model can be seen in Figure 4a ;like Figure 4a As shown in the figure, a cross-attention layer is added to the feature extraction module of each level in the Unet network to ensure that the original cross-attention layer in the feature extraction module is used to perform attention operations on the content description text, while the newly added cross-attention layer is used to perform attention operations on the features of the reference image embedded in the feature extraction module.

[0108] In the case where the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the system architecture of the image processing system can be seen in Figure 4b .like Figure 4b As shown, the image processing system at least includes a video generation model, input data of the video generation model (i.e., N latent space noise initialization features used when generating a frame sequence) and a second image processing module 302. The second image processing module 302 includes a new image encoder (expressed as image encoder) and a feature attenuation submodule provided in an embodiment of the present application. Among them, the image encoder can encode the reference image into the specified N image feature vectors to be attenuated embedding; and then use the cross-attention calculation in the feature extraction module of each level in the feature Unet network; so as to combine the content information of the reference image in the process of calling the video generation model for video generation.

[0109] It can be seen that this multi-level cross-attention embedding mechanism can pass the content information / features of the reference image to the process of video generation by the video generation model, thereby effectively ensuring that the image content of each video frame in the final generated target video is consistent with the image content of the reference image. In addition, a feature attenuation submodule is deployed in the first image processing module, so that the feature attenuation submodule can perform feature attenuation processing on the N image feature vectors to be attenuated extracted based on the reference image, thereby ensuring that the image content of each video frame has content differences with the image content of the reference image; further, the specific differences in the content differences between the image contents of the video frames are described by the content description text of the input video generation model, so as to ensure that the overall video content of the generated target video meets the video generation requirements and improve the video quality of the target video.

[0110] (3) The feature embedding mechanism is a two-stream consistent embedding mechanism (or called a two-stream embedding mechanism), which includes both the latent space embedding mechanism and the multi-level cross-attention embedding mechanism.

[0111] The dual-stream consistency embedding mechanism refers to the process of embedding the features of the reference image into the generation process of the frame sequence by using both the latent space embedding mechanism and the multi-level cross attention embedding mechanism at the model inference stage. For the relevant contents of the latent space embedding mechanism and the multi-level cross attention embedding mechanism, please refer to the above description, which will not be repeated here.

[0112] Considering that the two-stream consistency embedding mechanism includes a multi-level cross-attention embedding mechanism, it is necessary to Figure 2 The video generation model shown in FIG. 1 is fine-tuned in structure. The structural diagram of the fine-tuned video generation model is shown in the previous Figure 4a , the model structure of the fine-tuned video generation model is not described in detail here.

[0113] When the feature embedding mechanism is a two-stream consistent embedding mechanism, the system architecture of the image processing system can be seen in Figure 5 .like Figure 5 As shown, the image processing system at least includes a video generation model, input data of the video generation model (i.e., N latent space noise initialization features used when generating a frame sequence), a first image processing module 301, and a second image processing module 302. The composition and principle of the first image processing module 301 can be found in the aforementioned Figure 3 For the relevant description of the embodiment shown in FIG. 1 , similarly, for the relevant contents about the composition and principle of the second image processing module 302, please refer to the aforementioned Figure 4b The relevant description of the illustrated embodiment is not repeated here.

[0114] It can be seen that the innovative dual-stream consistency embedding mechanism designed in the embodiment of the present application is composed of two parts, namely, the latent space embedding mechanism and the multi-level cross-attention embedding mechanism. These two embedding mechanisms can give the features extracted from the reference image provided to the video generation model (or the designated information / consistency information used to guide the generation of video content, etc.) to the latent space noise initialization features used when generating the frame sequence (that is, the latent space produced in the initial stage of generation), and each level of the Unet network in the video generation model. Feature extraction module. Not only can the video generation model perceive the image content of the reference image from the dimension of the input data, but also can the video generation model perceive the image content of the reference image during the video generation process; this design that allows the video generation model to fully perceive the reference image to the main content information at multiple levels can effectively ensure that the image content of each frame of the generated video sequence frame can maintain a high degree of consistency with the image content of the currently provided reference image, so that the image content of the entire video sequence frame is more coherent and does not produce abrupt content.

[0115] It should be noted that, for the sake of ease of explanation, the image processing solution provided in the embodiment of the present application will be introduced in detail below using a feature embedding mechanism that is a dual-stream consistency embedding mechanism as an example.

[0116] In practical applications, the image processing system involved in the embodiment of the present application can be deployed to a single computer device, or distributed to multiple computer devices; in this way, after receiving the reference image and the content description text, the computer device can call the image processing system to use the image processing scheme to convert the reference image to the target video based on the reference image and the content description text. Specifically, the image processing system can be deployed directly in the computer device, or it can be deployed in an application, and the application runs in the computer device. An application may refer to a computer program for completing one or more specific tasks; by classifying the application according to different dimensions (such as the operation mode and function of the application), the type of the same application under different dimensions can be obtained. For example: according to the operation mode of the application, the application may include but is not limited to: a client installed in the terminal, a small program that can be used without downloading and installing (as a subprogram of the client), a global wide area network (World Wide Web, Web) application opened through a browser, etc. For another example: according to the functional type of the application, the application may include but is not limited to: an instant messaging (Instant Messaging, IM) application, a content interaction application, etc. Instant messaging applications refer to applications for instant messaging and social interaction based on the Internet. Instant messaging applications may include, but are not limited to: social applications with communication functions, map applications with social interaction functions, game applications, etc. Content interaction applications refer to applications that can realize content interaction, such as sharing platforms, personal spaces, news, videos, and app stores.

[0117] Among them, the computer device may include a terminal and a server. The terminal may include, but is not limited to: a smartphone (such as a smartphone deploying an Android system, or a smartphone deploying an Internet operating system (IOS)), a tablet computer, a portable personal computer, a vehicle-mounted device, a head-mounted device, a smart TV or a smart home device. The embodiment of the present application does not limit the type of terminal device, which is explained here. The server is a background server corresponding to the terminal, which is used to interact with the terminal to provide computing and application service support for the application deployed in the terminal. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0118] Furthermore, the image processing solution provided by the embodiment of the present application can be applied to any application scenario that requires image to video conversion; in the application scenario, for the video generation task of the video content that the user wants, by inputting a specified reference image, and then generating an entire video frame sequence, the main body of the image content of each frame of the video frame in the video frame sequence is consistent with the main body of the image content of the reference image, and at the same time, through the control of the feature attenuation mechanism, the main body of the image content of each frame of the video frame can show a certain difference from the main body of the image content of the reference image, ensuring that the overall video content change of the video frame sequence has a certain diversity. Exemplarily, the application scenarios may include but are not limited to: game application scenarios, camera application scenarios, and dynamic emoticon package generation scenarios (or called gif (Graphics Interchange Forma, graphics exchange format) emoticon packages), etc.; the following is a brief introduction to the process of implementing the image processing solution in these exemplary application scenarios; among which:

[0119] The application scenario is a game application scenario, specifically a game promotion material production scenario; in the game promotion material production scenario, it is often necessary to convert a game promotion image into a game promotion video (such as a game video of a game character in a game promotion image) so as to use the game promotion video to better promote the game. In this case, the computer device that deploys the image processing system can be a server at the game development end. Game developers can input a reference image of a game scene or a reference image containing a game character into the server, so that after the image processing system deployed in the server receives the reference image, the reference image is processed using the first image processing module and / or the second image processing module (such as image encoding, replication and feature attenuation processing, etc.) to obtain N attenuated image coding features; in this way, the N attenuated image coding features are embedded in the model reasoning stage of the video generation model (such as embedded in the N latent space noise initialization features used when generating frame sequences or the feature extraction modules of each level in the Unet network), and a target video with a game style and scene characters and other contents that are highly consistent and can have a variety of game scenes or game character micro-movements can be generated according to the reference image. In addition, according to the different attenuation strengths during feature attenuation processing, the changing speed of the video content in the generated target video can be controlled, thereby achieving the effect of producing diverse target videos for game promotion materials.

[0120] The application scenario is a camera application scenario, specifically an application scenario for converting camera images to videos; in this application scenario, the user may have a need to convert images that are taken or downloaded from the Internet into videos. In this case, the computer device is a terminal held by the user, in which a shooting application is running, and the image processing solution provided in the embodiment of the present application is deployed in the shooting application; in this way, after the user uses the shooting application in the terminal to shoot a reference image (or uploads a reference image through the shooting application), he can choose to generate a dynamic video live (i.e., a video stream transmitted in real time) for the reference image; at this time, the shooting application inputs the reference image into the image processing system, so that the image processing system converts the reference image according to the image processing solution provided in the embodiment of the present application to obtain the target video. Among them, the image content of each frame of the target video and the image content of the reference image have content consistency and content difference, thereby enhancing the user's experience of using the shooting application and improving the fun of the shooting application to a certain extent.

[0121] The application scenario is the dynamic expression package generation scenario, which is a GIF animation of a character's facial expression. In this case, the computer device can be a terminal held by the user, or a server held by the developer of the dynamic expression package, etc.; with the computer device as the terminal, an application for making expression packages is deployed in the terminal, then the user can upload a reference image in the application, such as a composite image of a comic strip or a famous scene in a film or TV series. In this way, the application inputs the reference image into the image processing system so that the reference image can be converted into a target video using an image processing solution. At this time, the target video is a dynamic expression package, and the image content of each video frame in the dynamic expression package is the character in the reference image, and the dynamic effect of playing multiple video frames is that the character performs micro-movements.

[0122] It should be noted that the above three application scenarios are examples of application scenarios to which the image processing solution provided in the embodiments of the present application is applicable, and do not limit the application scenarios to which the image processing solution is applicable.

[0123] To facilitate understanding of the application scenarios of the above-mentioned image processing solutions, Figure 6 An example of an application scenario in which an image processing solution is applicable is given. Figure 6As shown, the image processing system includes a terminal 601 and a server 602. The embodiment of the present application does not limit the number and naming of the terminal and the server. The terminal 601 and the server 602 can be directly or indirectly connected in a limited or wireless manner. The embodiment of the present application does not limit the communication method between devices. For example, the communication method between devices may include but is not limited to: using HTTP requests, remote procedure calls, sockets, and shared content.

[0124] The image processing solution provided in the embodiment of the present application can be executed by a computer device, which is equipped with the video generation model trained in the model training phase of the embodiment of the present application, and the first image processing module and / or the second image processing module that implement the image processing solution together with the video generation model. The computer device can be Figure 6 The terminal 601 and / or the server 602 in the system shown, that is, the embodiment of the present application supports the image processing scheme to be executed by any one of the terminal 601 and the server 602, or by the terminal 601 and the server 602 together. Below, taking the image processing scheme being executed by the terminal 601 and the server 602 together, and the first image processing module and the second image processing module being deployed in the terminal 601, and the video generation model being deployed in the server 602 as an example, an image-to-video application scenario applicable to the image processing scheme is described.

[0125] In a specific implementation, a user can upload or shoot a reference image through terminal 601, and set a content description text; the content description text is used to describe the image content of each frame of the target video that the user hopes to generate based on the reference image, such as the content description text is "a bear doing radio gymnastics", that is, the user hopes that the subject "bear" in the target video generated based on the reference image can perform radio gymnastics movements. After receiving the reference image, terminal 601 uses the first image processing module to perform image encoding and feature attenuation processing on the reference image to obtain N attenuated latent space reference features; at the same time, the second image processing module is used to perform image encoding and feature attenuation processing on the reference image to obtain N attenuated image feature vectors. Terminal 601 transmits the N attenuated latent space reference features, N attenuated image feature vectors and content description text to server 602. After receiving N attenuated latent space reference features, N attenuated image feature vectors and content description text, server 602 embeds N attenuated latent space reference features into N latent space noise initialization features used when generating frame sequences, obtains N latent space target features, and performs text recognition processing on the content description text to obtain text recognition vectors; in this way, server 602 also embeds N attenuated image encoding feature vectors into the feature extraction module of each level in the Unet network of the video generation model in the process of using text recognition vectors and N latent space target features to convert images into videos, so as to finally obtain the target video. Among them, the main body of the image content of each frame of the target video is "bear", which is consistent with the main body "bear" of the image content of the reference image, and the difference is that the expression form of "bear" in the target video, such as posture and expression, is consistent with the content description text, and is different from the expression form of "bear" in the image content of the reference image. In order to facilitate the user to obtain the converted target video in time, the server 602 also returns the generated target video to the terminal 601, so that the user can browse, copy, download and forward the target video from the terminal 601.

[0126] Based on the above brief introduction to the image processing solution and application architecture provided by the embodiment of the present application, the following points need to be explained:

[0127] ① The above-mentioned embodiments of the present application Figure 6 The application architecture shown is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. A person skilled in the art will know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems. In other words, Figure 6The schematic diagram of the image-to-video application architecture shown is an exemplary architecture schematic diagram; in actual applications, the number and distribution of devices (such as terminals and servers) included in the architecture may change, and the embodiments of the present application do not limit the architecture schematic diagram of the image-to-video application architecture.

[0128] ② The above is an introduction to the solution flow of the image processing solution by taking the first image processing module and the second image processing module deployed in the terminal 601 and the trained video generation model deployed in the server 602 as an example. In actual applications, depending on the different deployment locations of the first image processing module, the second image processing module and the video generation model, Figure 6 The scheme flow of implementing the image processing scheme of the illustrated architecture is different. For example, the first image processing module, the second image processing module and the video generation model are all deployed in the server; then after receiving the reference image and content description text uploaded by the user, the terminal 601 directly transmits the reference image and content description text to the server 602, and the server 602 uses the first image processing module and the second image processing module to perform operations such as image encoding and feature attenuation processing on the reference image. For another example, when only one feature embedding mechanism is adopted, the terminal 601 or the server 602 may only include the first image processing module or the second image processing module.

[0129] ③ The collection and processing of relevant data in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations. The acquisition of personal information requires the knowledge or consent of the individual subject (or the legal basis for obtaining the information), and the subsequent use and processing of data shall be carried out within the scope of authorization of laws and regulations and the subject of personal information. For example, when the embodiments of this application are applied to specific products or technologies, such as when a reference image is uploaded, it is necessary to obtain the permission or consent of the user who has the disposal authority for the reference image, and the collection, use and processing of relevant data (such as the conversion of reference images, etc.) shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0130] Based on the image processing scheme and system described above, the embodiment of the present application proposes a more detailed image processing method. The image processing method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0131] See also Figure 7 , Figure 7 It is a flow chart of an image processing method provided by an exemplary embodiment of the present application; the flow chart mainly provides a specific implementation process of converting image to video using a video generation model and a first image processing module from the perspective of model reasoning. Figure 7 The method flow shown can be executed by the aforementioned computer device, where the computer device is the aforementioned Figure 6The terminal 601 shown is used as an example for explanation. The image processing method may include but is not limited to steps S701-S704:

[0132] S701: Acquire a reference image and content description text.

[0133] As described above, the reference image is represented as ref image in the embodiment of the present application, which refers to a picture whose image content is used as the base content of the video content of the target video to be generated. When the reference image is introduced in the video generation process, the image content of the reference image can be used to provide guidance for the generation of the image content of each video frame in the frame sequence, thereby ensuring that the main body of the image content of each video frame in the generated target video (such as the main character and environment, etc.) is consistent with the main body of the image content of the reference image, that is, the image content of the reference image and the image content of each video frame in the frame sequence have content consistency. Similarly, the content description text is a string used to describe the image content of each video frame in the target video to be generated. When the content description text is introduced in the video generation process, the content description text can be used to provide detailed content guidance for the image content of each video frame in the target video frame, such as the content details of the image content of the video frame or the content trend of the video content of the entire frame sequence. It should be noted that the reference image and the content description text are both used to generate the same target video, so it is necessary to ensure that there is no semantic conflict between the image content of the reference image and the image content described by the content description text; that is, the main content of the image content described by the content description text needs to be consistent with the main content of the image content of the reference image. For example, if the main content of the reference image is a standing bear and the content description text is "dancing girl", then there will be a semantic conflict between the subject "bear" of the reference image and the subject "girl" of the image content described by the content description text.

[0134] It can be seen that in the process of using reference images and content description texts to generate target videos, the role of the reference image is to ensure that the main content of each video frame in the generated target video and the image content of the reference image are consistent with each other, and the role of the content description text is to ensure that the expression form of the main content of each video frame in the generated target video matches the image content described by the content description text. In this way, it can effectively ensure that each video frame in the target video is generated based on the image content of the reference image, and the content description text can be used to guide the generation of the difference between the image content of each video frame in the target video and the image content of the reference image, thereby ensuring that the entire video content change of the target video is consistent with the image content described by the content description text.

[0135] According to different application scenarios of the image processing method provided in the embodiment of the present application, the providers of reference images and content description texts are different, such as the providers can be users or developers, etc., and the number of providers can be one or more. For example: Assume that the image processing method is applied to a camera application scenario. In this scenario, the user has the need to convert images into videos. In this case, the user needs to upload the reference image and input the content description text through a computer device. For example, the camera application scenario is provided by a camera software deployed in the terminal, such as Figure 8 As shown, a page in the camera software may be provided with a button for converting an image to a video, such as button 801; when button 801 is triggered, an upload page 802 is displayed, and the upload page 802 includes an image upload area 8021 and a text upload area 8022; thus, the user can upload images taken historically or downloaded from the Internet as reference images in the image upload area 8021, and manually input or copy and paste text as content description text in the text upload area 8022. The present application embodiment does not limit the source and acquisition method of the reference image and the content description text, but is specifically described here.

[0136] S702: Perform coding replication processing on the reference image to obtain N coding features of the image to be attenuated.

[0137] In order to use the image content of a reference image to guide the generation of image content of N video frames in a target video; after acquiring the reference image, the embodiment of the present application supports encoding and copying the reference image to obtain N image coding features to be attenuated extracted from the reference image, and the N image coding features to be attenuated are all the same, and any image coding feature to be attenuated can be understood as an image feature of the reference image (such as color features, texture features, shape features, spatial relationship features, etc.), which can be used to characterize the content characteristics of the image content of the reference image.

[0138] In a specific implementation, after obtaining a reference image, the reference image is first subjected to image coding processing to obtain coding features of the image to be attenuated, which is intended to extract the coding features of the image to be attenuated from the reference image, which are used to characterize the content features / content characteristics of the image content of the reference image. Then, considering that the target video to be generated is a frame sequence consisting of N video frames, in order to facilitate the generation of the image content of each video frame based on the image content of the reference image, it is also necessary to copy the coding features of the image to be attenuated N times to obtain N coding features of the image to be attenuated.

[0139] As mentioned above, the embodiments of the present application provide two feature embedding mechanisms, namely, latent space embedding mechanism and multi-level cross-attention embedding mechanism. Different feature embedding mechanisms correspond to different attenuated image coding features, that is, different feature embedding mechanisms need to use different image coding processing methods to perform image coding processing on the reference image to ensure that the image coding features to be attenuated extracted by the image coding processing of the reference image can be embedded in the video generation model.

[0140] In the embodiment of the present application, the image coding processing method corresponding to the latent space embedding mechanism is introduced. When the feature embedding mechanism is the latent space embedding mechanism, the image coding feature to be attenuated corresponding to the latent space embedding mechanism can be called the latent space reference feature to be attenuated, and the latent space reference feature to be attenuated can be expressed as reflatent or ref feature. Among them, when the feature embedding mechanism is the latent space embedding mechanism, the system framework diagram provided in the embodiment of the present application can be referred to as the aforementioned Figure 3 .like Figure 3 As shown, the first image processing module 301 included in the system framework includes the encoder part of the variational autoencoder of the video generation model (Stable Diffusion model), namely, the VAE encoder; in this way, the variational autoencoder of the video generation model can be used to perform image encoding processing on the reference image, aiming to encode the reference image into the reference feature of the latent space to be attenuated required by the video generation model. Among them, the principle of the variational autoencoder performing image encoding processing on the reference image can include: the variational autoencoder receives the input reference image, and extracts the image features of the reference image through a series of convolutional layers or other feature extraction algorithms, and the image features can be used to characterize the key information in the reference image. The variational autoencoder then maps the image features of the reference image to the latent space to obtain the reference features of the latent space to be attenuated that characterize the content characteristics of the image content of the reference image in the latent space; by mapping the image features of the reference image to the latent space, these image features can be used to learn more latent features of the reference image, so as to deepen the understanding of the reference image, thereby facilitating the subsequent guidance of the generation of video content based on more correct and rich image features, and improving the quality of the video content.

[0141] Further, use Figure 3The variational autoencoder in the system framework shown performs image encoding processing on the reference image, and after obtaining the reference feature of the latent space to be attenuated, the reference feature of the latent space to be attenuated is directly copied into N copies, that is, N reference features of the latent space to be attenuated are obtained; one reference feature of the latent space to be attenuated corresponds to a frame of video in the target video to be generated, thereby ensuring that the video content of each frame of video in the target video is consistent with the image content of the reference image.

[0142] It is worth noting that the embodiment of the present application chooses to use the variational autoencoder of the video generation model to perform image encoding processing on the reference image, mainly based on the following considerations: ① Figure 1a From the system framework of the Stable Diffusion model in the model training stage, it can be seen that the massive sample data used in the model training stage are all obtained by outputting the image to the variational autoencoder, specifically the encoder vae encoder, which is encoded by the encoder vae encoder and then adding noise. Therefore, the embodiment of the present application supports the use of the encoder vae encoder provided by the video generation model to perform image encoding processing on the reference image, which can ensure that the feature semantics extracted from the reference image can better adapt to the training semantics of the Stable Diffusion model.

[0143] ② Considering that the features calculated for the reference image (i.e., the aforementioned latent space reference features to be attenuated, specifically, the attenuated latent space reference features after feature attenuation processing of the latent space reference features to be attenuated) need to be embedded in the Stable Diffusion model obtained by the model training mentioned in ① above, specifically, embedded in the latent space noise initialization features (noise latent) used when the Stable Diffusion model is used to generate a frame sequence, it is necessary to ensure that the feature size of the latent space reference features to be attenuated extracted for the reference image is consistent with the feature size of the latent space noise initialization feature noise latent. Therefore, the embodiment of the present application designs the use of the variational autoencoder trained by the Stable Diffusion model in the model training phase to perform image encoding processing on the reference image, which can ensure that the feature size of the latent space reference features to be attenuated extracted for the reference image is consistent with the feature size of the latent space noise initialization feature noiselatent, that is, to ensure that the latent space reference features to be attenuated obtained by performing image encoding processing on the reference image in the model inference phase can meet the size requirements of the Stable Diffusion model for input data.

[0144] ③From the above description, it can be seen that the Stable Diffusion model will use a large number of images to train the encoder vaeencoder in the model training stage, so the encoder vae encoder trained by the model is able to completely extract the main content semantics in the image; therefore, the embodiment of the present application uses the encoder vae encoder trained by the Stable Diffusion model in the model training stage to encode the reference image, which can better extract the main content semantics in the reference image and improve the accuracy of feature extraction.

[0145] S703: Perform feature attenuation processing on the N to-be-attenuated image coding features to obtain N attenuated image coding features.

[0146] Based on the above description, after the reference image is coded and N to-be-attenuated image coding features are obtained, if the N identical to-be-attenuated image coding features are directly used to guide the generation of the image content of N video frames in the target video, then the image content of the generated N video frames will be the same; for example, N=16, if the original consistency information (i.e., the to-be-attenuated image coding features) extracted from the reference image is used in the generation process of 16 video frames, then all 16 video frames generated using the original to-be-attenuated image coding features will be exactly the same. However, in fact, according to the video playback order, the image content of adjacent video frames in the N video frames of the target video is different in content, so as to ensure that the entire video content is dynamically changing, such as when a person's posture changes or the scene changes.

[0147] In order to achieve flexible adjustment of the image content of N video frames, the embodiment of the present application supports the use of a feature attenuation strategy to perform feature attenuation processing on N image coding features to be attenuated extracted from a reference image, and uses the N attenuated image coding features after feature attenuation to guide the generation of N video frames. Among them, the feature attenuation strategy can be understood as a strategy for gradually weakening the features of N identical image coding features to be attenuated from a feature dimension. In this way, after the N image coding features to be attenuated are processed by feature attenuation, the N attenuated image coding features obtained are not the same, and the feature difference between the N attenuated image coding features and the image coding features to be attenuated is also increasing. Therefore, when guiding the generation of N video frames based on the N attenuated image coding features, it can be ensured that there is a content difference between the image content of the N video frames and the image content of the reference image.

[0148] The main principle of the feature attenuation strategy provided in the embodiment of the present application is: set a corresponding attenuation coefficient for each video frame in N video frames, and use the attenuation coefficient to perform feature attenuation processing on the image coding features to be attenuated that are referenced in the generation process of the corresponding video frames. In detail, it supports pre-setting an attenuation parameter, and the value range of the attenuation parameter is a decimal between (0, 1); therefore, the smaller the value of the attenuation parameter, the greater the content difference between adjacent video frames, so that the speed of change of the video content of the target video is greater; it can be seen that the attenuation parameter can be used to indicate the degree of content difference between the image content of adjacent video frames in the frame sequence. Then, the pre-set attenuation parameters are used to calculate N attenuation coefficients, and the N attenuation coefficients correspond to the N image coding features to be attenuated one by one; the so-called one-to-one correspondence means that one attenuation coefficient is used as the feature weight of an image coding feature to be attenuated. Finally, the N attenuation coefficients are respectively multiplied by the corresponding image coding features to be attenuated to obtain N attenuated image coding features.

[0149] For ease of understanding, the following Fig. 9 The characteristic attenuation process described above is described in detail. Fig. 9 As shown, assuming that the attenuation parameter is expressed as The default attenuation coefficient of the first image coding feature to be attenuated is 1.0, and the attenuation coefficient of the second image coding feature to be attenuated is (Right now ), the attenuation coefficient corresponding to the third image coding feature to be attenuated is (Right now ), the attenuation coefficient corresponding to the fourth image coding feature to be attenuated is (Right now ), and so on, the attenuation coefficient corresponding to the Nth image coding feature to be attenuated is (Right now ). Then, the first image coding feature to be attenuated among the N image coding features to be attenuated is multiplied by its corresponding attenuation coefficient 1.0 to obtain the first attenuated image coding feature, which is the same as the first image coding feature to be attenuated, that is, the image content of the first video frame in the N video frames is completely generated according to the image content of the reference image; the second image coding feature to be attenuated and its corresponding attenuation coefficient w are multiplied by the second image coding feature to be attenuated and its corresponding attenuation coefficient w 1A product operation is performed to obtain a second attenuated image coding feature. Since w is a decimal in the range of 0 to 1, the second attenuated image coding feature is different from the second image coding feature to be attenuated. The second attenuated image coding feature attenuates the content semantic information of the second image coding feature to be attenuated, that is, the image content of the second video frame in the N video frames is generated only according to part of the image content of the reference image; and so on, the attenuated image coding feature corresponding to each of the N image coding features to be attenuated is calculated.

[0150] It should be noted that ① the embodiment of the present application does not limit the specific value of the attenuation parameter; the content change speed between frames is different according to the different values ​​of the attenuation parameter. For example, the larger the value of the attenuation parameter, the faster the content change speed between frames, that is, the greater the content difference between the image content of the video frame and the image content of the reference image. In this way, by controlling the intensity of frame-by-frame attenuation (that is, the value of the attenuation parameter), the content change speed of the video content of the target video can be effectively controlled, thereby enriching the content change diversity of the video frame sequence.

[0151] ② The present application embodiment does not limit the calculation rules for calculating the attenuation coefficient based on the attenuation parameter. Fig. 9 The calculation rule is introduced by taking the power of the attenuation parameter as the attenuation coefficient as an example. In practical applications, the calculation rule can also include the square root of the attenuation coefficient as the attenuation coefficient, etc. For example, assuming that the attenuation parameter is a positive number greater than 1, the attenuation coefficient corresponding to the first image coding feature to be attenuated among the N image coding features to be attenuated is 1.0 by default, and the attenuation coefficient corresponding to the second image coding feature to be attenuated is , the attenuation coefficient corresponding to the third image coding feature to be attenuated is , and so on, the attenuation coefficient corresponding to the Nth image coding feature to be attenuated.

[0152] S704: Perform video generation processing on the N attenuated image coding features according to the image content described in the content description text to obtain a target video.

[0153] Based on the aforementioned step S703, after calculating the N attenuated image coding features, the embodiment of the present application supports embedding the N attenuated image coding features into the video generation process, so that the generation of the video content of the target video can be guided by the image content of the reference image, thereby ensuring that the image content of the reference image and the image content of each video frame in the target video have consistency. In a specific implementation, the computer device obtains N latent space noise initialization features (noise latent), which are obtained by adding random noise to a blank latent space reference feature; in short, the latent space noise initialization feature can be simply understood as being obtained by adding random noise to a blank image, so as to facilitate processing of the random noise in the latent space noise initialization feature to draw a video frame; considering that a latent space noise initialization feature is used to generate a video frame in a frame sequence, it is necessary to copy the latent space noise initialization feature to obtain N latent space noise initialization features. And, the computer device calls the text encoding module text encoder to perform text recognition processing on the content description text to obtain a text recognition vector, which is used to represent the semantic information of the content description text. Furthermore, the video generation model is called to perform video generation processing on the text recognition vector and N latent space noise initialization features, and N attenuated image coding features are embedded in the process of video generation processing to obtain the target video.

[0154] As described above, the present embodiment provides two feature embedding mechanisms, namely, latent space feature embedding mechanism and multi-level cross attention embedding mechanism. Figure 3 The system framework shown in the figure introduces the specific implementation process of calling the video generation model to perform video generation processing on the text recognition vector and N latent space noise initialization features when the feature embedding mechanism is the latent space feature embedding mechanism, and embedding N attenuated image coding features in the process of video generation processing to obtain the target video. Figure 3As shown, the image coding feature to be attenuated is represented as the latent space reference feature to be attenuated, and the image coding feature after attenuation is represented as the latent space reference feature after attenuation. In this case, before inputting the N latent space noise initialization features into the video generation model, the N latent space reference features after attenuation are first embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set, which includes N latent space target features, and one latent space target feature is obtained by embedding an attenuated latent space reference feature into the corresponding latent space noise initialization feature; that is, the N latent space reference features after attenuation are respectively embedded with the corresponding latent space noise initialization features in the N latent space noise initialization features to obtain a latent space reference feature set. In this way, when the video generation model is called to perform video generation processing on the latent space reference feature set and the text recognition vector, it is possible to embed the consistency information of the reference image into the generation process of each video frame when the frame sequence is generated, thereby ensuring that the image content of each video frame in the obtained target video refers to the image content of the reference image.

[0155] Among them, the above-mentioned embedding of N attenuated latent space reference features into corresponding latent space noise initialization features respectively means: performing point-wise addition operation on N attenuated latent space reference features and corresponding latent space noise initialization features respectively. In other words, performing point-wise addition operation on N attenuated image coding features and corresponding latent space noise initialization features respectively, a latent space reference feature set is obtained; the latent space reference feature set includes N latent space target features obtained by point-wise addition operation, and one latent space target feature is obtained by performing point-wise addition operation on attenuated image coding features and corresponding latent space noise initialization features. Exemplarily, a schematic diagram of performing point-wise addition operation on N attenuated latent space reference features and corresponding latent space noise initialization features respectively can be seen. Fig.10 .like Fig.10 As shown, the point-wise addition operation of the attenuated image coding feature and the corresponding latent space noise initialization feature refers to: adding the pixels at the same position in the attenuated image coding feature with the same feature size and the corresponding latent space noise initialization feature to generate a new latent space target feature.

[0156] In summary, on the one hand, the embodiments of the present application support feature attenuation processing of the reference features of the latent space to be attenuated extracted from the reference image, which helps the video generation model to gradually reduce the reference to the reference image in the process of generating N frames of video frames, and can introduce more image content described by the content description text to control the dynamic changes of the video content. The control of the attenuation coefficient can achieve the diversity of video content. On the other hand, the embodiments of the present application support the effective control of the content change speed of the video content of the target video that is finally generated by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), thereby enriching the content change diversity of the video frame sequence. On the other hand, an embodiment of the present application supports embedding consistency information extracted from a reference image into a video generation process. When the feature embedding mechanism is a latent space embedding mechanism, the attenuated latent space reference feature extracted from the reference image can be embedded into the latent space noise initialization feature used in frame sequence generation before video generation; thereby directly passing the main content semantic information in the reference image to the generation process of the image content of each video frame when generating the frame sequence, so as to guide the generation of the image content of each video frame, thereby ensuring the specific content consistency between the image content of the video frame and the image content of the reference image; and, guiding the content difference between the image content of the video frame and the image content of the reference image through the image content described in the content description text, thereby helping users obtain a target video that can match the image content described in the content description text and has content consistency with the image content of the reference image.

[0157] See also Fig.11 , Fig.11 It is a flow chart of another image processing method provided by an exemplary embodiment of the present application; the flow chart mainly provides a specific implementation process of converting image to video using a video generation model and a second image processing module from the perspective of model reasoning. Fig.11 The method flow shown can be executed by the aforementioned computer device, where the computer device is the aforementioned Figure 6 The terminal 601 shown is used as an example for explanation. The image processing method may include but is not limited to steps S1101-S1106:

[0158] S1101: Acquire a reference image and content description text.

[0159] It should be noted that the specific implementation process of the embodiment shown in step S1101 is the same as the above Figure 7 The specific implementation process shown in step S701 in the illustrated embodiment is the same, and reference may be made to the relevant description of the specific implementation process shown in the aforementioned step S701, which will not be repeated here.

[0160] S1102: Perform image coding processing on the reference image to obtain coding features of the image to be attenuated.

[0161] S1103: Copy the image coding features to be attenuated N times to obtain N image coding features to be attenuated.

[0162] In steps S1102-S1103, as described above, the feature embedding mechanism provided by the embodiment of the present application includes: a latent space embedding mechanism and a multi-level cross attention embedding mechanism. When the feature embedding mechanism is a latent space embedding mechanism, the specific implementation process shown in steps S1102-S1103 can be referred to the above Figure 7 The relevant description of the relevant paragraphs shown in step S702 in the illustrated embodiment is not repeated here. In the embodiment of the present application, the specific implementation process of performing image copying processing on the reference image (i.e., the specific implementation process shown in steps S1102-S1103) is mainly introduced when the feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0163] Specifically, when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the specific implementation process shown in steps S1102-S1103 is as follows: Figure 4b The second image processing module shown is executed. The second image processing module includes: an image encoder (represented as image encoder) and a feature attenuation submodule. Among them, the image encoder is used to perform image encoding processing on the reference image to obtain the coding features of the image to be attenuated, specifically the feature vector of the image to be attenuated (represented as image embedding); the feature attenuation submodule is used to perform feature attenuation processing on the feature vector of the image to be attenuated encoded by the image encoder to obtain the attenuated image feature vector. Furthermore, the attenuated image feature vector output by the feature attenuation submodule will be input into each level feature extraction module in the Unet network, so that the entire Unet network can perceive the semantic information of the main content in the reference image at multiple levels, thereby improving the image content of the reference image and the image content of each video frame in the frame sequence. The consistency of the content.

[0164] The image encoder used in the multi-level cross attention embedding mechanism of the embodiment of the present application can refer to the process of image encoding processing for the reference image. Fig.12 ;like Fig.12 As shown, the second image processing module performs image coding processing on the reference image to obtain the feature vector of the image to be attenuated, and copies the feature vector of the image to be attenuated, which may include but is not limited to steps s11-s13; wherein:

[0165] s11: Perform image preprocessing on the reference image to obtain the initial image feature vector.

[0166] like Fig.13 As shown, the image encoder included in the second image processing module includes a preprocessing encoding module, which can be represented as a clip image encoder (which can be abbreviated as a clip model). As described above, the clip model can fully recognize elements such as objects, scenes, actions, etc. in the image, and can also understand text related to the image, such as tags, descriptions, titles, etc.; therefore, the embodiment of the present application sets the clip model in the image encoder, which can achieve full and accurate recognition of the content semantic information in the reference image by virtue of the good recognition performance of the clip model.

[0167] Specifically, after the reference image is output to the image encoder, the preprocessing encoding module in the image encoder is used to preprocess the reference image to obtain an initial image feature vector. The initial image feature vector includes a feature vector of the image to be attenuated and preprocessing information; wherein the feature vector of the image to be attenuated is used to represent the main content semantic information in the reference image, and the preprocessing information indicates the image analysis information for the reference image, and the image analysis information mainly includes the semantic information of the function of the clip model itself, such as the classification prediction information of the clip model for the content in the reference image.

[0168] s12: Perform content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated.

[0169] After the initial image feature vector is extracted from the reference image based on the aforementioned step s11, it is necessary to perform content extraction processing (or content purification processing) on ​​the initial image feature vector to extract the image feature vector to be attenuated, which is used to characterize the main content semantic information of the reference image, from the initial image feature vector. In an embodiment of the present application, a preset initial image semantic vector is used to assist in the content purification of the initial image feature vector; the initial image semantic vector can be represented as an initial image embedding, which can be used to indicate the weight distribution of content extraction processing from the initial image semantic vector, and the initial image semantic vector can be initialized using random weights. Specifically, after obtaining the preset initial image semantic vector, the initial image feature vector is subjected to content extraction processing based on the initial image semantic vector to obtain the image feature vector to be attenuated. See. Fig.13The content extraction processing here includes attention operation and full connection operation. Specifically, the preset initial image semantic vector and the initial image feature vector encoded by the clip image encoder are input into the cross attention layer for attention operation, and then the cross attention layer is input into a fully connected layer (Multilayer Perceptron, MLP) to integrate the features obtained by the cross attention layer to purify the image feature vector to be attenuated from the initial image feature vector.

[0170] Furthermore, the embodiment of the present application sets up a multiple content purification mechanism to implement multiple content extraction processing of the initial image feature vector, aiming to improve the feature purity of the to-be-attenuated image feature vector extracted from the initial image feature vector. Fig.13 As shown, the content extraction process is performed by a content extraction module. Assume that the number of the content extraction modules is M, where M is a positive integer; one content extraction module corresponds to one content extraction process / content purification process, and each content extraction module includes a cross attention layer and a fully connected layer. Based on this, after the first content extraction module performs content extraction on the initial image feature vector and the initial image semantic vector, the output information of the first content extraction module can be input into the second content extraction module to continue the content extraction process, and so on, the output information of the last content extraction module is used as the image feature vector to be attenuated.

[0171] In the process of multiple content extraction processes described above, in order to ensure that the original content semantics of the reference image will not be lost too much during the multiple content extraction processes, the embodiment of the present application is also designed to add the input information of the cross attention layer in the same content extraction module to the output information of the fully connected layer as the input information of the next content extraction module, so as to ensure that each content extraction process does not lose too much original content semantics of the reference image. Fig.13 As shown, assuming that the number of content extraction modules is M, when M=1, the input information of the first content extraction module, that is, the input information processed by the first content extraction process, is: the initial image semantic vector and the initial image feature vector. When M=2, the input information of the second content extraction module, that is, the input information processed by the second content extraction process, is: the output information of the first content extraction module (that is, the processing result obtained by the first content extraction process) and the splicing information of the initial image semantic vector (that is, the information obtained by splicing the features or vectors). When M>2, the input information of the Mth content extraction module, that is, the input information processed by the Mth content extraction process, is: the splicing information of the output information of the M-1th content extraction module and the output information of the M-2th content extraction module.

[0172] s13: Copy the feature vector of the image to be attenuated N times to obtain N feature vectors of the image to be attenuated.

[0173] After extracting a pure image feature vector to be attenuated from the initial image feature vector, it is necessary to copy the image feature vector to be attenuated into N copies according to the number of frames N of the target video to be generated, so as to obtain N image feature vectors to be attenuated.

[0174] In summary, based on the specific implementation process shown in the aforementioned steps s11-s13, the feature vector of the image to be attenuated can be calculated from the reference image. In the case of a multi-level cross-attention embedding mechanism, the consideration of using an image encoder to perform image encoding processing on the reference image in the embodiment of the present application is to ensure that the feature vector of the image to be attenuated encoded by the image encoder can be embedded in the feature extraction module of each level in the Unet network in the video generation model. In other words, the data format of the feature vector of the image to be attenuated after image encoding processing using the image encoder meets the data format requirements of the feature extraction module for the input data.

[0175] S1104: Perform feature attenuation processing on the N to-be-attenuated image coding features to obtain N attenuated image coding features.

[0176] It should be noted that the specific implementation process shown in step S1104 and Figure 7 The specific implementation process shown in step S703 in the illustrated embodiment is the same, and reference may be made to the relevant description of the specific implementation process shown in step S703, which will not be repeated here.

[0177] S1105: Obtain N latent space noise initialization features, and perform text recognition processing on the content description text to obtain a text recognition vector.

[0178] S1106: Calling the video generation model to perform video generation processing on the text recognition vector and the N latent space noise initialization features, and embedding N attenuated image coding features during the video generation processing to obtain a target video.

[0179] In steps S1105-S1106, as described above, the feature embedding mechanism provided by the embodiment of the present application includes: a latent space embedding mechanism and a multi-level cross attention embedding mechanism. When the feature embedding mechanism is a latent space embedding mechanism, the specific implementation process shown in steps S1105-S1106 can be referred to the above Figure 7The relevant description of the relevant paragraphs shown in step S704 in the illustrated embodiment is not repeated here. In the embodiment of the present application, the specific implementation process of embedding the attenuated image feature vector into the video generation model (i.e., the specific implementation process shown in steps S1105-S1106) is mainly introduced when the feature embedding mechanism is a multi-level cross-attention embedding mechanism.

[0180] Specifically, when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the specific implementation process shown in steps S1105-S1106 can be referred to Fig.14 .like Fig.14 As shown, the embodiment of the present application fine-tunes the feature extraction module (Unet block) in the Unet network to obtain a fine-tuned feature extraction module (new Unet block). Specifically, the fine-tuned feature extraction module adds a cross-attention mechanism corresponding to the text. In this way, the fine-tuned feature extraction module includes a first attention submodule corresponding to the text and a second attention submodule corresponding to the image. In other words, compared to Figure 1b With regard to the feature extraction module before fine-tuning shown in the figure, a second attention submodule (which integrates cross attention) is mainly added to the feature extraction module after fine-tuning. The second attention submodule is used to receive the attenuated image feature vector extracted by the second image processing module for the reference image, and after performing attention operation on the attenuated image feature vector, the attention operation result and the attention result obtained by performing attention operation on the text recognition vector are added together, so that the main content information in the reference image can be input into the entire video generation process. It should be noted that when the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the feature extraction modules mentioned in the embodiments of the present application all refer to the feature extraction modules after fine-tuning, that is, Fig.14 The feature extraction module shown.

[0181] based on Fig.14In the fine-tuned feature extraction module shown, the embedding method of the N attenuated image feature vectors output by the second image processing module into the video generation process may include: in the process of video generation processing, embedding the N attenuated image feature vectors into the feature extraction module of each level in the Unet network in the video generation model; then, calling the feature extraction module to perform video generation processing on the embedded N attenuated image feature vectors, text recognition vectors and N latent space noise initialization features to obtain the target video. Among them, taking any level feature extraction module in the multi-level feature extraction module as the target feature extraction module as an example, the process of embedding the N attenuated image feature vectors into the target feature extraction module is introduced; specifically: through the second attention mechanism in the target feature extraction module, the received N attenuated image feature vectors are subjected to attention operation to obtain the second attention operation result; and through the first attention mechanism in the target feature extraction module, the received text is subjected to attention operation to obtain the first attention operation result; and then the second attention operation result and the first attention operation result are fused to obtain the output information of the target feature extraction module. Through the above embedding process, the main content information in the reference image can be embedded into the video generation process, specifically embedded into the feature extraction module at each level, so as to improve the content consistency between the image content of each frame of the target video finally generated and the image content of the reference image.

[0182] For more details, see Fig.14 It can be seen that when the target feature extraction module receives N attenuated image feature vectors, it also receives the text recognition vector obtained for content description text recognition, as well as the preceding features (e.g., the preceding features received by the first feature extraction module are: N latent space noise initialization features, and the preceding features received by the other feature extraction modules except the first feature extraction module among the M feature extraction modules are: the feature information after the previous feature extraction module adjacent to the feature extraction module performs feature extraction processing on the N latent space noise initialization features). Therefore, the process in which the target feature extraction module uses the second attention submodule to perform attention operation on the N attenuated image feature vectors specifically includes: using the second attention submodule to perform attention operation on the N attenuated image feature vectors and the preceding features to obtain the second attention operation result; similarly, the process in which the target feature extraction module uses the first attention submodule to perform attention operation on the text recognition vector specifically includes: using the first attention submodule to perform attention operation on the text recognition vector and the preceding features to obtain the first attention operation result.

[0183] In summary, on the one hand, the embodiments of the present application support feature attenuation processing of the reference features of the latent space to be attenuated extracted from the reference image, which helps the video generation model to gradually reduce the reference to the reference image in the process of generating N frames of video frames, and can introduce more image content described by the content description text to control the dynamic changes of the video content. The control of the attenuation coefficient can achieve the diversity of video content. On the other hand, the embodiments of the present application support the effective control of the content change speed of the video content of the target video that is finally generated by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), thereby enriching the content change diversity of the video frame sequence. On the other hand, the embodiments of the present application support embedding the consistency information extracted from the reference image into the video generation process. When the feature embedding mechanism is a multi-level cross-attention embedding mechanism, the attenuated image feature vector extracted from the reference image can be embedded in the attention operation process of the feature extraction module at each level in the Unet network, so that the image content of each video frame of the Unet model matches the main content semantic information in the reference image when generating a frame sequence, so that the video content of the entire video frame sequence and the image content of the reference image achieve a high degree of content consistency, ensuring that the video content of the entire video frame sequence is more coherent and does not produce content abruptness; and, the image content described in the content description text is used to guide the content difference between the image content of the video frame and the image content of the reference image, thereby helping users obtain a target video that can match the image content described in the content description text and has content consistency with the image content of the reference image.

[0184] See also Fig.15 , Fig.15 It is a flow chart of another image processing method provided by an exemplary embodiment of the present application; the flow chart mainly provides a specific implementation process of converting image to video using a video generation model, a first image processing module and a second image processing module from the perspective of model reasoning. Fig.15 The method flow shown can be executed by the aforementioned computer device, such as the aforementioned computer device. Figure 6 The terminal 601 and / or server 602 shown. The image processing method may include but is not limited to steps S1501-S1507:

[0185] S1501: Acquire a reference image and content description text.

[0186] It should be noted that the specific implementation process shown in step S1501 is the same as the above Figure 7 The specific implementation process shown in step S701 in the illustrated embodiment is the same, and reference may be made to the relevant description of the specific implementation process shown in the aforementioned step S701, which will not be repeated here.

[0187] S1502: Map the reference image to a latent space to obtain a latent space reference feature to be attenuated that represents content characteristics of the image content of the reference image in the latent space.

[0188] It should be noted that the specific implementation process shown in step S1502 can be found in the aforementioned Figure 7 The relevant descriptions of the relevant paragraphs shown in step S702 in the illustrated embodiment are not repeated here.

[0189] S1503: performing image preprocessing on the reference image to obtain an initial image feature vector, and performing content extraction processing on the initial image feature vector to obtain a feature vector of the image to be attenuated.

[0190] It should be noted that the specific implementation process shown in step S1503 can be found in the aforementioned Fig.11 The relevant descriptions of the relevant paragraphs shown in step s11 in the illustrated embodiment are not repeated here.

[0191] The embodiment of the present application does not limit the execution order of step S1502 and step S1503.

[0192] S1504: Copy the image coding features to be attenuated N times to obtain N image coding features to be attenuated, where the N image coding features to be attenuated include a latent space reference feature to be attenuated and an image feature vector to be attenuated.

[0193] It should be noted that the specific implementation process shown in step S1504 is the same as the above Figure 7 The specific implementation process shown in step S703 in the illustrated embodiment is the same, and reference may be made to the relevant description of the specific implementation process shown in step S703, which will not be repeated here.

[0194] S1505: embed N attenuated latent space reference features into corresponding latent space noise initialization features respectively to obtain a latent space reference feature set, and input the latent space reference feature set into a video generation model.

[0195] It should be noted that the specific implementation process shown in step S1505 can be found in the aforementioned Figure 7 The relevant descriptions of the relevant paragraphs shown in step S704 in the illustrated embodiment are not repeated here.

[0196] S1506: Embed N attenuated image feature vectors into each level feature extraction module.

[0197] It should be noted that the specific implementation process shown in step S1506 can be found in the aforementioned Fig.11 The relevant descriptions of the relevant paragraphs shown in step S1106 in the illustrated embodiment are not repeated here.

[0198] The embodiment of the present application does not limit the execution order of step S1505 and step S1506.

[0199] S1507: Generate a target video based on the embedded N attenuated image coding features, where the N to-be-attenuated image coding features include a to-be-attenuated latent space reference feature and a to-be-attenuated image feature vector.

[0200] In a specific implementation, when the feature embedding mechanism includes both a latent space embedding mechanism and a multi-level cross-attention embedding mechanism, generating a target video based on the embedded N attenuated image coding features may include: calling a video generation model to perform video generation processing on a text recognition vector, a latent space reference feature set, and N attenuated image feature vectors to obtain a target video. It can be seen that when the feature embedding mechanism includes both a latent space embedding mechanism and a multi-level cross-attention embedding mechanism, the generation process of the target video will simultaneously refer to the N attenuated latent space reference features output by the first image processing module and the N attenuated image feature vectors output by the second image processing module; wherein, the embedding method of embedding the N attenuated latent space reference features and the N attenuated image feature vectors into the video generation model can refer to the aforementioned related description and will not be repeated here.

[0201] In summary, on the one hand, the embodiment of the present application supports feature attenuation processing of the latent space reference features to be attenuated extracted from the reference image, helping the video generation model to gradually reduce the reference to the reference image in the process of generating N frames of video frames, and can introduce more image content described by the content description text to control the dynamic changes of the video content. The control of the attenuation coefficient can achieve the diversity of video content. On the other hand, the embodiment of the present application supports effectively controlling the content change speed of the video content of the target video that is finally generated by controlling the intensity of frame-by-frame attenuation (i.e., the value of the attenuation parameter), thereby enriching the content change diversity of the video frame sequence. On the other hand, the embodiment of the present application supports two feature embedding mechanisms at the same time. The latent space embedding mechanism directly passes the main content semantic information in the reference image to the generation process of the image content of each video frame when generating the frame sequence, which is used to guide the generation of the image content of each video frame. The multi-level cross-attention embedding mechanism can embed the attenuated image feature vector extracted from the reference image into the attention operation process of each level of the feature extraction module in the Unet network, so that the image content of each video frame in the Unet model matches the main content semantic information in the reference image when generating the frame sequence (that is, the image content of the video frame is generated with reference to the main content semantic information of the reference image), so that the video content of the entire video frame sequence and the image content of the reference image reach a high degree of content consistency, ensuring that the video content of the entire video frame sequence is more coherent and does not produce abrupt content. Therefore, through the combination of the two feature embedding mechanisms, the video generation model can fully receive the main content semantic information in the reference image, and improve the content consistency between the image content of each video frame in the final generated target video and the image content of the reference image. On the other hand, the image content described by the content description text guides the content difference between the image content of the video frame and the image content of the reference image, thereby helping users obtain a target video that can match the image content described by the content description text and has content consistency with the image content of the reference image.

[0202] The above detailed description of the method of the embodiment of the present application, in order to facilitate the above-mentioned scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly. In the embodiment of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that contains the function of the module or unit.

[0203] Fig.16 A schematic diagram of the structure of an image processing device provided by an exemplary embodiment of the present application is shown; the image processing device can be used to perform Figure 7 , Fig.11 or Fig.15 Some or all of the steps in the method embodiment shown. Fig.16 , the device comprises the following units:

[0204] The acquisition unit 1601 is used to acquire a reference image and a content description text; the reference image is used to generate a target video, and the target video includes a frame sequence consisting of N video frames; the content description text is used to describe the image content that each video frame in the frame sequence should contain; N is a positive integer;

[0205] The processing unit 1602 is used to perform coding replication processing on the reference image to obtain N coding features of the image to be attenuated, where the coding features of the image to be attenuated are used to characterize content characteristics of the image content of the reference image;

[0206] The processing unit 1602 is further used to perform feature attenuation processing on the N to-be-attenuated image coding features to obtain N attenuated image coding features; each attenuated image coding feature is used to generate a video frame in the frame sequence; the feature attenuation processing is used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image;

[0207] The processing unit 1602 is also used to call the video generation model to perform video generation processing on N attenuated image coding features according to the image content described by the content description text to obtain a target video; the image content of each video frame in the frame sequence included in the target video matches the image content described by the content description text, and has content differences with the image content of the reference image.

[0208] In one implementation, the processing unit 1602 is configured to perform feature attenuation processing on N to-be-attenuated image coding features to obtain N attenuated image coding features, specifically for:

[0209] Obtaining a preset attenuation parameter, where the attenuation parameter is used to indicate a degree of content difference between image contents of adjacent video frames in a frame sequence;

[0210] The attenuation parameters are used to calculate N attenuation coefficients; the N attenuation coefficients correspond one to one to the N encoding features of the image to be attenuated;

[0211] The N attenuation coefficients are multiplied by the corresponding coding features of the image to be attenuated to obtain N coding features of the attenuated image.

[0212] In one implementation, the processing unit 1602 is used to perform coding replication processing on the reference image to obtain N coding features of the image to be attenuated, specifically to:

[0213] Performing image coding processing on the reference image to obtain coding features of the image to be attenuated;

[0214] The coding feature of the image to be attenuated is copied N times to obtain N coding features of the image to be attenuated.

[0215] In one implementation, the coding feature of the image to be attenuated includes a reference feature of the latent space to be attenuated. The processing unit 1602 is configured to perform image coding processing on the reference image to obtain the coding feature of the image to be attenuated, specifically for:

[0216] The reference image is mapped to the latent space to obtain the reference features of the latent space to be attenuated.

[0217] In one implementation, the coding feature of the image to be attenuated includes a feature vector of the image to be attenuated. The processing unit 1602 is configured to perform image coding processing on the reference image to obtain the coding feature of the image to be attenuated, specifically for:

[0218] Performing image preprocessing on the reference image to obtain an initial image feature vector; the initial image feature vector includes a feature vector of the image to be attenuated and preprocessing information, and the preprocessing information indicates image analysis information for the reference image;

[0219] The initial image feature vector is subjected to content extraction processing to obtain the feature vector of the image to be attenuated.

[0220] In one implementation, the processing unit 1602 is used to perform content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated, specifically to:

[0221] Obtaining a preset initial image semantic vector, where the initial image semantic vector is used to indicate a weight distribution for content extraction processing from the initial image feature vector;

[0222] Based on the initial image semantic vector, the initial image feature vector is subjected to content extraction processing to obtain the feature vector of the image to be attenuated; the content extraction processing includes an attention operation and a full connection operation.

[0223] In one implementation, the content extraction process is performed by a content extraction module. The number of content extraction modules is M. One content extraction module corresponds to one content extraction process. M is a positive integer. The input information of the content extraction module includes:

[0224] When M=1, the input information of the first content extraction module is: initial image semantic vector and initial image feature vector;

[0225] When M=2, the input information of the second content extraction module is: the output information of the first content extraction module and the concatenation information of the initial image semantic vector;

[0226] When M>2, the input information of the Mth content extraction module is: the concatenated information of the output information of the M-1th content extraction module and the output information of the M-2th content extraction module.

[0227] In one implementation, the processing unit 1602 is used to perform video generation processing on N attenuated image coding features according to the image content described by the content description text to obtain the target video, specifically for:

[0228] Obtaining N latent space noise initialization features, where the latent space noise initialization features are obtained by adding random noise to a blank latent space reference feature; one latent space noise initialization feature is used to generate a video frame in a frame sequence; and,

[0229] Perform text recognition processing on the content description text to obtain a text recognition vector; the text recognition vector is used to represent the semantic information of the content description text;

[0230] The video generation model is called to perform video generation processing on the text recognition vector and N latent space noise initialization features, and N attenuated image coding features are embedded in the video generation process to obtain the target video.

[0231] In one implementation, the attenuated image coding features include attenuated latent space reference features, and the processing unit 1602 is used to call the video generation model to perform video generation processing on the text recognition vector and the N latent space noise initialization features, and embed the N attenuated image coding features in the process of video generation processing to obtain the target video, specifically for:

[0232] The N attenuated latent space reference features are respectively embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set; the latent space reference feature set includes N latent space target features, and a latent space target feature is obtained by embedding an attenuated latent space reference feature into a corresponding latent space noise initialization feature;

[0233] The video generation model is called to perform video generation processing on the latent space reference feature set and the text recognition vector to obtain the target video.

[0234] In one implementation, the video generation model includes a multi-level feature extraction module;

[0235] The attenuated image coding features include attenuated image feature vectors. The processing unit 1602 is used to embed N attenuated image coding features during the video generation process, specifically for:

[0236] During the video generation process, N attenuated image feature vectors are embedded into each level feature extraction module in the video generation model.

[0237] In one implementation, the feature extraction module includes a first attention submodule corresponding to the text and a second attention submodule corresponding to the image; any level feature extraction module in the multi-level feature extraction module is represented as a target feature extraction module; the processing unit 1602 is used to embed N attenuated image feature vectors into each level feature extraction module in the video generation model, specifically for:

[0238] Performing an attention operation on the N attenuated image feature vectors through a second attention submodule in the target feature extraction module to obtain a second attention operation result; and

[0239] Performing an attention operation on the text recognition vector through the first attention submodule in the target feature extraction module to obtain a first attention operation result;

[0240] The second attention operation result and the first attention operation result are fused to obtain the output information of the target feature extraction module.

[0241] In one implementation, the video generation model includes multiple layers of feature extraction modules, each layer of feature extraction module is embedded with a spatiotemporal attention module; the processing unit 1602 is further used to:

[0242] The spatiotemporal attention module is used to predict the output information output by the corresponding feature extraction module to obtain a prediction result; the prediction result is used to indicate: when a frame sequence is generated based on the output information output by the corresponding feature extraction module, the content correlation between the image contents of two adjacent video frames.

[0243] According to one embodiment of the present application, Fig.16The various units in the image processing device shown can be individually or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, it can be executed by running on a general computing device such as a computer including processing elements and storage elements such as a central processing unit (CPU), a random access memory (RAM), and a read-only memory (ROM). Figure 7 , Fig.11 or Fig.15 A computer program (including program code) for each step involved in the corresponding method shown in FIG. Fig.16 The image processing apparatus shown in the figure can be used to implement the image processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0244] In the embodiment of the present application, on the one hand, since the N attenuated image coding features are obtained based on the N image coding features to be attenuated, and the N image coding features to be attenuated indicate the content characteristics of the image content of the reference image, when the video generation model is called to perform video generation processing on the N attenuated image coding features to obtain the target video, the N attenuated image coding features are relied upon to characterize the content characteristics of the image content of the reference image from different feature degrees, so that the image content of each video frame in the frame sequence included in the target video generated based on the N attenuated image coding features has content consistency with the image content of the reference image, such as the subject in the image (such as a person, environment, etc.) appears as the same subject, and at the same time, there is a content difference between the image content and the reference image. On the other hand, in the process of calling the generation model to perform video generation processing on N attenuated image coding features to obtain the target video, the reference to the reference image can be reduced frame by frame in the process of calling the video generation model to generate N video frames according to the reference image, helping the video generation model to introduce more image content described by the content description text to constrain / guide the generation of the target video; not only can it be ensured that the image content of each video frame in the frame sequence included in the target video frame matches the image content described by the content description text, such as the expression form of the subject in the video frame matches (such as the action of a person matches the action described by the content description text), but also the image content described by the content description text can assist in the generation of image elements of the content difference part between the image content of the video frame and the image content of the reference image. On the other hand, each of the N attenuated image coding features is used to generate a video frame. Since the N attenuated image coding features are not the same, it can be ensured that the image content of the N video frames is also different, thereby ensuring that the target video can present the effect of content change.

[0245] Fig.17 FIG. 1 shows a schematic diagram of a computer device provided by an exemplary embodiment of the present application. Fig.17 , the computer device includes a processor 1701, a communication interface 1702 and a computer-readable storage medium 1703. The processor 1701, the communication interface 1702 and the computer-readable storage medium 1703 may be connected via a bus or other means. The communication interface 1702 is used to receive and send data. The computer-readable storage medium 1703 may be stored in the memory of the computer device, the computer-readable storage medium 1703 is used to store computer programs, and the processor 1701 is used to execute the computer programs stored in the computer-readable storage medium 1703. The processor 1701 (or CPU) is the computing core and control core of the computer device, which is suitable for implementing one or more computer programs, and is specifically suitable for loading and executing one or more computer programs to implement corresponding method flows or corresponding functions.

[0246] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, one or more computer programs suitable for being loaded and executed by the processor 1701 are also stored in the storage space. These computer programs can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0247] In one embodiment, the computer device may be the terminal or server mentioned in the aforementioned embodiment; one or more computer programs are stored in the computer-readable storage medium; the processor 1701 loads and executes the one or more computer programs stored in the computer-readable storage medium to implement the corresponding steps in the above-mentioned image processing method embodiment; in a specific implementation, the one or more computer programs in the computer-readable storage medium are loaded by the processor 1701 and execute the steps of each embodiment of the present application; wherein, the steps of each embodiment of the present application can refer to the relevant description of the aforementioned embodiments, and will not be repeated here.

[0248] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the image processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0249] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned image processing method is implemented.

[0250] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0251] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs (one or more). When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer program can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media integrated. Available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, digital video disks (DVDs)), or semiconductor media (eg, solid state disks (SSDs)).

[0252] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: include: Obtain reference images and content description text; The reference image is used to generate a target video, wherein the target video includes a frame sequence consisting of N video frames; The content description text is used to describe the image content that each video frame in the frame sequence should contain; N is a positive integer; Performing encoding and duplication processing on the reference image to obtain N encoding features of the image to be attenuated, wherein the encoding features of the image to be attenuated are used to characterize content characteristics of the image content of the reference image; Acquire an attenuation coefficient corresponding to each of the image coding features to be attenuated, and perform feature attenuation processing on the corresponding image coding features to be attenuated among the N image coding features to be attenuated according to the attenuation coefficient corresponding to each of the image coding features to be attenuated, so as to obtain N attenuated image coding features; each of the attenuated image coding features is used to generate a video frame in the frame sequence; the feature attenuation processing is used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image; the attenuation coefficient corresponding to any of the image coding features to be attenuated is calculated according to a preset attenuation parameter, and the value of the attenuation parameter is negatively correlated with the degree of content difference between the image contents of adjacent video frames in the frame sequence; According to the image content described by the content description text, video generation processing is performed on the N attenuated image coding features to obtain the target video; the image content of each video frame in the frame sequence included in the target video matches the image content described by the content description text, and has content differences with the image content of the reference image.

2. The method according to claim 1, characterized in that The performing feature attenuation processing on the N image coding features to be attenuated to obtain N attenuated image coding features includes: Acquire a preset attenuation parameter, where the attenuation parameter is used to indicate a degree of content difference between image contents of adjacent video frames in the frame sequence; The attenuation parameters are used to calculate N attenuation coefficients; the N attenuation coefficients correspond one to one with the N encoding features of the image to be attenuated; The N attenuation coefficients are respectively multiplied by the corresponding image coding features to be attenuated to obtain N attenuated image coding features.

3. The method according to claim 1, characterized in that The encoding and duplication processing is performed on the reference image to obtain N encoding features of the image to be attenuated, including: Performing image coding processing on the reference image to obtain coding features of the image to be attenuated; The image coding feature to be attenuated is copied N times to obtain N image coding features to be attenuated.

4. The method according to claim 3, characterized in that The coding feature of the image to be attenuated includes a reference feature of a latent space to be attenuated, and performing image coding processing on the reference image to obtain the coding feature of the image to be attenuated includes: The reference image is mapped to a latent space to obtain a reference feature of the latent space to be attenuated.

5. The method according to claim 3 or 4, characterized in that The coding feature of the image to be attenuated includes a feature vector of the image to be attenuated, and the image coding process is performed on the reference image to obtain the coding feature of the image to be attenuated, including: Performing image preprocessing on the reference image to obtain an initial image feature vector; the initial image feature vector includes the feature vector of the image to be attenuated and preprocessing information, and the preprocessing information indicates image analysis information for the reference image; Content extraction processing is performed on the initial image feature vector to obtain the feature vector of the image to be attenuated.

6. The method according to claim 5, characterized in that The performing content extraction processing on the initial image feature vector to obtain the feature vector of the image to be attenuated includes: Acquire a preset initial image semantic vector, where the initial image semantic vector is used to indicate a weight distribution for content extraction processing from the initial image feature vector; Based on the initial image semantic vector, content extraction processing is performed on the initial image feature vector to obtain the feature vector of the image to be attenuated; the content extraction processing includes attention operation and full connection operation.

7. The method according to claim 6, characterized in that The content extraction process is performed by a content extraction module, the number of the content extraction modules is M, one content extraction module corresponds to one content extraction process, and M is a positive integer; The input information of the content extraction module includes: When M=1, the input information of the first content extraction module is: the initial image semantic vector and the initial image feature vector; When M=2, the input information of the second content extraction module is: the output information of the first content extraction module and the concatenation information of the initial image semantic vector; When M>2, the input information of the Mth content extraction module is: the concatenated information of the output information of the M-1th content extraction module and the output information of the M-2th content extraction module.

8. The method according to claim 1, characterized in that The step of performing video generation processing on the N attenuated image coding features according to the image content described by the content description text to obtain the target video includes: Acquire N latent space noise initialization features, wherein the latent space noise initialization features are obtained by adding random noise to a blank latent space reference feature; one of the latent space noise initialization features is used to generate a video frame in the frame sequence; and, Performing text recognition processing on the content description text to obtain a text recognition vector; the text recognition vector is used to represent the semantic information of the content description text; The video generation model is called to perform video generation processing on the text recognition vector and the N latent space noise initialization features, and the N attenuated image coding features are embedded in the video generation process to obtain the target video.

9. The method according to claim 8, characterized in that The attenuated image coding features include attenuated latent space reference features, the calling of the video generation model performs video generation processing on the text recognition vector and the N latent space noise initialization features, and embedding the N attenuated image coding features in the process of the video generation processing to obtain the target video, including: The N attenuated latent space reference features are respectively embedded into the corresponding latent space noise initialization features to obtain a latent space reference feature set; the latent space reference feature set includes N latent space target features, and one latent space target feature is obtained by embedding one of the attenuated latent space reference features into the corresponding latent space noise initialization feature; The video generation model is called to perform video generation processing on the latent space reference feature set and the text recognition vector to obtain the target video.

10. The method according to claim 8 or 9, characterized in that The video generation model includes a multi-level feature extraction module; The attenuated image coding feature includes an attenuated image feature vector, and embedding N attenuated image coding features in the process of the video generation process includes: During the video generation process, the N attenuated image feature vectors are embedded into the feature extraction module at each level in the video generation model.

11. The method according to claim 10, characterized in that The feature extraction module includes a first attention submodule corresponding to the text and a second attention submodule corresponding to the image; any level of the feature extraction modules in the multi-level feature extraction modules is represented as a target feature extraction module; The step of embedding the N attenuated image feature vectors into the feature extraction module at each level in the video generation model comprises: Performing an attention operation on the N attenuated image feature vectors through the second attention submodule in the target feature extraction module to obtain a second attention operation result; and Performing an attention operation on the text recognition vector through the first attention submodule in the target feature extraction module to obtain a first attention operation result; The second attention operation result and the first attention operation result are integrated to obtain the output information of the target feature extraction module.

12. The method according to claim 8, characterized in that The video generation model includes multiple layers of feature extraction modules, each layer of the feature extraction module is embedded with a spatiotemporal attention module; the method further includes: The spatiotemporal attention module is used to perform prediction processing on the output information output by the corresponding feature extraction module to obtain a prediction result; the prediction result is used to indicate: when the frame sequence is generated based on the output information output by the corresponding feature extraction module, the content correlation between the image content of two adjacent video frames.

13. An image processing device, characterized in that: include: An acquisition unit, used for acquiring a reference image and a content description text; The reference image is used to generate a target video, and the target video includes a frame sequence consisting of N video frames; the content description text is used to describe the image content that each video frame in the frame sequence should contain; N is a positive integer; A processing unit, configured to perform coding and copying processing on the reference image to obtain N coding features of the image to be attenuated, wherein the coding features of the image to be attenuated are used to characterize content characteristics of the image content of the reference image; The processing unit is further used to obtain an attenuation coefficient corresponding to each of the image coding features to be attenuated, and perform feature attenuation processing on the corresponding image coding features to be attenuated among the N image coding features to be attenuated according to the attenuation coefficient corresponding to each of the image coding features to be attenuated, so as to obtain N attenuated image coding features; each of the attenuated image coding features is used to generate a video frame in the frame sequence; the feature attenuation processing is used to control the content difference between the image content of each video frame in the frame sequence and the image content of the reference image; the attenuation coefficient corresponding to any of the image coding features to be attenuated is calculated according to a preset attenuation parameter, and the value of the attenuation parameter is negatively correlated with the degree of content difference between the image contents of adjacent video frames in the frame sequence; The processing unit is further used to perform video generation processing on the N attenuated image coding features according to the image content described by the content description text to obtain the target video; the image content of each video frame in the frame sequence included in the target video matches the image content described by the content description text, and has content differences with the image content of the reference image.

14. A computer device, characterized in that: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 12 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the image processing method according to any one of claims 1 to 12.

16. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the image processing method according to any one of claims 1 to 12 is implemented.

Images