Video generation method and device, computer equipment and storage medium

Abstract

The invention discloses a video generation method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of inputting a to-be-processed image into an image processing model to obtain a first image feature; inputting the to-be-processed image into a global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a combined image feature; based on the combined image features, using a first video generation model to generate an initial video corresponding to the to-be-processed image; obtaining video text description, and encoding the video text description to obtain text encoding features; and inputting the initial video and the text coding feature into a second video generation model to obtain a target video corresponding to the to-be-processed image. The invention also relates to the technical field of block chains, and the to-be-processed image is stored in a block chain network. According to the method, visual details can be reproduced, global consistency can be kept, and the visual quality and semantic consistency of the video are effectively improved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a video generation method, apparatus, computer equipment and storage medium. Background Art

[0002] AI video generation technology is the process of automatically creating or editing video content using artificial intelligence technology. This technology achieves automatic generation and personalized customization of video content through the comprehensive application of multiple AI technologies such as deep learning, computer vision, and natural language processing.

[0003] AI video generation technology has made significant progress in recent years and has shown broad application prospects in various fields. With the continuous development of deep learning and generative models, video generation technology has gradually become one of the hot spots in the field of artificial intelligence. At present, video generation models are mainly implemented based on the CLIP (Control Language-Image Pre-training) model. The training goal of the CLIP model is to align visual features and language features. Although the CLIP model can enable the model to learn high-level feature representations, this method ignores the perception of details in the image, resulting in tasks that require precise visual information recognition or image generation. The CLIP model may not be able to accurately capture and reproduce subtle differences in the image, such as texture, color gradients, or small objects in complex scenes. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a video generation method, apparatus, computer device and storage medium to solve the technical problem that the existing video generation model easily ignores the perception of details in the image, resulting in the model being unable to accurately capture and reproduce subtle differences in the image in tasks requiring precise visual information recognition or image generation.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a video generation method, which adopts the following technical solution:

[0006] A video generation method, comprising:

[0007] Acquire an image to be processed, and input the image to be processed into a pre-trained image processing model to obtain a first image feature;

[0008] Inputting the image to be processed into a pre-trained global encoder to obtain a second image feature;

[0009] Combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature;

[0010] Based on the first combined image feature, using a pre-trained first video generation model to generate an initial video corresponding to the image to be processed, wherein the first video generation model is a pre-trained video generation diffusion model;

[0011] Obtaining a video text description, and encoding the video text description to obtain a text encoding feature;

[0012] The initial video and text encoding features are input into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

[0013] Furthermore, the image processing model is used to capture local features of the image. The image processing model is built based on the CLIP model architecture. The CLIP model architecture includes an image encoder and a text encoder. The image to be processed is obtained, and the image to be processed is input into the pre-trained image processing model to obtain the first image feature. The steps specifically include:

[0014] Perform text extraction on the image to be processed to obtain image text;

[0015] Use an image encoder to encode the image to be processed to obtain image coding features;

[0016] Use a text encoder to encode the image text to obtain image text encoding features;

[0017] Mapping the image coding features and the image text coding features to a preset first feature space, and aligning the image coding features and the image text coding features in the first feature space;

[0018] A feature alignment result of the first feature space is obtained to obtain a first image feature.

[0019] Furthermore, the step of inputting the image to be processed into a pre-trained global encoder to obtain a second image feature specifically includes:

[0020] Perform image division on the image to be processed to obtain a number of image blocks;

[0021] Use the global encoder to encode each image block in turn to obtain image block features;

[0022] Each image block feature is sequentially mapped to a preset second feature space, and the feature association relationship between each image block feature is captured in the second feature space to obtain a second image feature.

[0023] Furthermore, the step of combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature specifically includes:

[0024] Inputting the first image feature and the second image feature into a pre-trained attention weight model, and outputting an attention weight value;

[0025] Dimensionally aligning the first image feature and the second image feature;

[0026] The attention weight value is used to perform weighted summation on the first image feature and the second image feature after the dimension alignment is completed to obtain a first combined image feature.

[0027] Furthermore, the step of dimensionally aligning the first image feature and the second image feature specifically includes:

[0028] Generate adaptive scaling factors based on the needs of the video generation task;

[0029] Add adaptive scaling factors to the attention weight model;

[0030] Based on the adaptive scaling factor, the first image feature and the second image feature are dimensionally aligned in the attention weight model.

[0031] Furthermore, based on the first combined image feature, the step of using the pre-trained first video generation model to generate an initial video corresponding to the image to be processed specifically includes:

[0032] Using a first video generation model, add video noise to the first combined image feature to obtain a first noisy video, wherein the video noise is Gaussian noise;

[0033] Using a preset denoising function to denoise the first noisy video to obtain an initial video key frame;

[0034] The initial video key frames are combined to obtain the initial video corresponding to the image to be processed.

[0035] Furthermore, the step of inputting the initial video and text encoding features into the pre-trained second video generation model to obtain the target video corresponding to the image to be processed specifically includes:

[0036] Obtaining an initial video key frame, and performing feature extraction on the initial video key frame to obtain key frame features;

[0037] Combining the key frame feature and the text encoding feature to obtain a second combined image feature;

[0038] Performing an upsampling operation on the second combined image feature, and converting the resolution of the upsampled feature into a target resolution, wherein the target resolution is a required resolution of the video generation task;

[0039] Adding video noise to the upsampled features after the resolution conversion to obtain a second noisy video;

[0040] Use a denoising function to denoise the second noisy video to obtain a target video key frame;

[0041] Combine the target video key frames to obtain the target video corresponding to the image to be processed.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a video generation device, which adopts the following technical solution:

[0043] A video generating device, comprising:

[0044] A first feature extraction module is used to obtain an image to be processed and input the image to be processed into a pre-trained image processing model to obtain a first image feature;

[0045] A second feature extraction module, used for inputting the image to be processed into a pre-trained global encoder to obtain a second image feature;

[0046] A feature weight combination module, used to combine the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature;

[0047] A first video generation module, configured to generate an initial video corresponding to the image to be processed based on the first combined image feature using a pre-trained first video generation model, wherein the first video generation model is a pre-trained video generation diffusion model;

[0048] The third feature extraction module is used to obtain the video text description and encode the video text description to obtain text encoding features;

[0049] The second video generation module is used to input the initial video and text encoding features into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

[0050] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0051] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of any one of the above-mentioned video generation methods when executing the computer-readable instructions.

[0052] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0053] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method described in any one of the above are implemented.

[0054] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0055] The present application discloses a video generation method, device, computer equipment and storage medium, belonging to the field of artificial intelligence technology. The present application obtains a first image feature by inputting the image to be processed into a pre-trained image processing model; inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a first combined image feature; based on the first combined image feature, using the pre-trained first video generation model to generate an initial video corresponding to the image to be processed; obtaining a video text description, and encoding the video text description to obtain a text encoding feature; inputting the initial video and the text encoding feature into the pre-trained second video generation model to obtain a target video corresponding to the image to be processed. The present application extracts the first image feature through the image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through the global encoder, and the second image feature captures the overall features of the image. The fusion of the two features ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0057] Figure 1 An exemplary system architecture diagram is shown in which the present application can be applied;

[0058] Figure 2 A flow chart showing an embodiment of a video generation method according to the present application;

[0059] Figure 3 A schematic structural diagram of an embodiment of a video generating device according to the present application is shown;

[0060] Figure 4 A schematic structural diagram of an embodiment of a computer device according to the present application is shown. DETAILED DESCRIPTION

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0062] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0065] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0066] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0067] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0068] It should be noted that the video generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the video generation device is generally arranged in the server / terminal device.

[0069] It should be understood that Figure 1 The numbers of terminal devices, networks and servers in the embodiment are merely illustrative. The above system may have any number of terminal devices, networks and servers according to implementation requirements.

[0070] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for video generation according to the present application. The video generation method comprises the following steps:

[0071] S201, obtaining an image to be processed, and inputting the image to be processed into a pre-trained image processing model to obtain a first image feature.

[0072] Specifically, the first image feature focuses on local details, which usually include low-level visual features such as texture, edge, shape, etc. of the image, for example, the color, texture pattern, or detail changes of a specific area. This feature can accurately capture the microscopic structural information of the image, which is very important for reproducing the details, texture and local shape of the image.

[0073] The process of extracting the first image feature is mainly achieved through the image encoder of the CLIP model. The CLIP model aims to map images and texts into a shared vector space and is trained using contrastive learning so that semantically similar image and text features are close in space.

[0074] In this step, first obtain the image to be processed for generating a video, that is, the source image. The source image can be a static image or a frame in a dynamic sequence. Then, use a pre-trained image processing model to extract features from the image to obtain the first image feature. The image processing model here can be a multi-modal model such as CLIP, which has a strong visual understanding ability and can capture details, edge information, local features, etc. in the image. These features contain important local detail information in the image, such as texture, color distribution, etc. Through this pre-trained model, accurate visual features can be obtained as one of the important inputs for subsequent video generation.

[0075] S202: Input the image to be processed into a pre-trained global encoder to obtain a second image feature.

[0076] Specifically, the second image feature focuses on global semantics and context, that is, it captures high-level abstract features from the overall perspective of the image. The first image feature focuses on a single pixel or a small area, while the second image feature focuses on the layout of the entire image, the relationship between objects, or the global semantic concept. The second image feature can provide overall semantic information, allowing the model to understand the global structure and overall layout of the image, such as the relative positions of different objects, the overall picture of the scene, etc.

[0077] The main purpose of this step is to understand the overall structure of the image from a global perspective. The global encoder is different from the image processing model. The global encoder pays more attention to the macro information of the image, such as the overall outline of the object, the layout, the relationship between the background and the foreground, etc. The pre-trained global encoder can be a model of the self-attention mechanism (such as the Transformer architecture). This type of model can capture the correlation between different regions in the image and ensure that the generated content has a reasonable spatial distribution as a whole. By extracting global features, the model can understand the global context of the image and supplement the semantic information that may be missed by local features.

[0078] S203: Combine the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature.

[0079] If we only rely on local features, the generated video may be accurate in details, but the overall structure or semantics may be wrong. On the contrary, if we only rely on global features, although the video may look good overall, the details may be distorted. Therefore, combining these two features helps to generate videos that are both detailed and globally reasonable.

[0080] Specifically, the first image feature and the second image feature provide different levels of information. Local features focus on details and ensure that the generated video contains delicate image information (such as texture and edge), while global features help the model understand the overall structure or semantics of the image and ensure that the generated video is consistent in global semantics.

[0081] In this step, the model will fuse the features extracted from local and global features based on a specific feature weighting algorithm to generate a comprehensive image feature vector. The weighting algorithm can be adjusted according to the task requirements. For example, in scenes with high detail requirements, local features may be given higher weights, while in complex scenes, global features may be given higher weights. This fusion process ensures that the video generation model can take into account both the details and the global structure of the image, so that the generated content is both detailed and overall harmonious. Common combinations include simple linear weighting, adaptive weight adjustment through attention mechanisms, or nonlinear combination of features using deep networks.

[0082] S204: Based on the first combined image feature, use a pre-trained first video generation model to generate an initial video corresponding to the image to be processed, wherein the first video generation model is a pre-trained video generation diffusion model.

[0083] Specifically, the system uses a pre-trained diffusion model to generate the initial video. The diffusion model is a type of generative model based on progressive denoising, which can iteratively generate a sequence that matches the image features from the input random noise. The diffusion model is conditioned on the combined image features and guides the denoising process so that the generated video gradually matches the content and style of the input image. The initial video is a low-resolution version, but it already contains the basic visual information of the image. The diffusion model generates each frame by gradually adjusting the noise distribution and ensuring the temporal continuity between frames.

[0084] S205, obtaining a video text description, and encoding the video text description to obtain a text encoding feature.

[0085] Specifically, the system obtains text descriptions related to the images to be processed and the generated videos, and encodes them. The text description can be a short video script or label provided by the user, such as "a beach scene in the sun" or "a mountain view at sunset". Then, a pre-trained text encoding model (such as CLIP or BERT) is used to convert these natural language descriptions into high-dimensional text feature vectors. These text features capture the semantic information in the description and help the model understand the content requirements and style tendencies of video generation. The text encoding process can capture keywords, scene descriptions, emotions or visual effects in the text. Through these features, the video generation model can further adjust the generation results to make the video content more in line with the user's intentions or specific contexts.

[0086] S206, inputting the initial video and text encoding features into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

[0087] Specifically, the pre-trained second video generation model receives two inputs: the initial video and the text encoding features. The initial video contains basic image content and motion information, while the text encoding features provide semantic constraints and guidance. The role of the second video generation model is to reconstruct and optimize the initial video at high resolution, refine the video details through super-resolution technology, and ensure that the content is consistent with the text description. The model can be based on GAN (Generative Adversarial Network) or Transformer architecture to enhance the clarity, details and visual consistency of the video frame by frame. In this process, the model uses text encoding features to adjust the generation results, such as optimizing the color, light and shadow effects in the video according to the style, scene or emotion of the description. In the end, the generated target video is not only high-resolution and rich in details, but also accurately reflects the input image and text description, making the video content more vivid and realistic.

[0088] In the above embodiment, the present application extracts the first image feature through the image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through the global encoder, and the second image feature captures the overall characteristics of the image. The fusion of the two features ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video.

[0089] Furthermore, the image processing model is used to capture local features of the image. The image processing model is built based on the CLIP model architecture. The CLIP model architecture includes an image encoder and a text encoder. The image to be processed is obtained, and the image to be processed is input into the pre-trained image processing model to obtain the first image feature. The steps specifically include:

[0090] Perform text extraction on the image to be processed to obtain image text;

[0091] Encode the image to be processed using an image encoder to obtain image coding features;

[0092] Use a text encoder to encode the image text to obtain image text encoding features;

[0093] Mapping the image coding features and the image text coding features to a preset first feature space, and aligning the image coding features and the image text coding features in the first feature space;

[0094] A feature alignment result of the first feature space is obtained to obtain a first image feature.

[0095] In this embodiment, firstly, the image to be processed is subjected to text extraction, and the relevant text content is automatically identified from the image, which can be realized by OCR optical character recognition technology to extract the text information in the image. Next, the image itself is encoded using a pre-trained image encoder to extract the visual features of the image, which include information such as the color, texture, and structure of the image. As for the extracted text, it is encoded by a text encoder to obtain the semantic features corresponding to the text, ensuring that the text information in the image can also be fully utilized. In order to enable the image features and text features to be compared and integrated under the same framework, the image encoding features and the image text encoding features are mapped to this unified first feature space. In the first feature space, a specific alignment method is used so that the features of the image and the text can be correlated and aligned with each other, thereby ensuring the semantic consistency of the two in the same space. Through feature alignment, the visual features of the image and the semantic features of the text complement each other, and finally generate a more informative first image feature, which can better express the overall semantics and local details of the image.

[0096] Through the above steps, by mapping the image encoding features and the image text encoding features into a unified feature space and performing feature alignment, the feature expression capability is improved in the cross-modal alignment of image and text, ensuring that the generated video or image has more semantic consistency and visual accuracy.

[0097] Furthermore, the step of inputting the image to be processed into a pre-trained global encoder to obtain a second image feature specifically includes:

[0098] Perform image division on the image to be processed to obtain a number of image blocks;

[0099] Use the global encoder to encode each image block in turn to obtain image block features;

[0100] Each image block feature is sequentially mapped to a preset second feature space, and the feature association relationship between each image block feature is captured in the second feature space to obtain a second image feature.

[0101] In this embodiment, the image to be processed is first divided into images, that is, the entire image is disassembled into several smaller image blocks. Next, each image block is encoded in turn using a global encoder to extract the features of each image block. The global encoder can adopt an architecture similar to a convolutional neural network (CNN) or a Transformer, which is good at capturing the global structural features of the image while retaining local details. Each image block is encoded to obtain its unique feature representation, which may contain information such as color, texture, shape, etc. in the block. After that, the features of each image block are mapped to a preset second feature space, which is used to uniformly manage and understand these image block features. In the second feature space, the system captures the association between the features of each image block, revealing the context, positional relationship and global layout between the image blocks. This feature association can be achieved through attention mechanisms or graph neural networks (GNNs), etc., to ensure that the model can not only understand the independent information of each image block, but also obtain the interaction and dependency between image blocks. Finally, the integrated second image features can fully express the overall structure and semantic information of the image.

[0102] Through the above steps, by dividing the image, encoding each image block separately, and capturing the association between image blocks in the feature space, the model's ability to understand the global structure and local details of the image is successfully enhanced. This not only improves the model's processing effect on complex images, but also effectively maintains the global consistency of the image and the association between local features.

[0103] Furthermore, the step of combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature specifically includes:

[0104] Inputting the first image feature and the second image feature into a pre-trained attention weight model, and outputting an attention weight value;

[0105] Dimensionally aligning the first image feature and the second image feature;

[0106] The attention weight value is used to perform weighted summation on the first image feature and the second image feature after the dimension alignment is completed to obtain a first combined image feature.

[0107] In this embodiment, the first image feature and the second image feature are first input into a pre-trained attention weight model, and the function of the attention weight model is to generate corresponding weight values ​​according to the importance of the image feature. The attention mechanism can effectively improve the model's ability to capture key details in complex scenes by calculating the relative importance of each feature. The first image feature usually represents a local feature, such as a detailed texture, while the second image feature may reflect more global information, such as the overall layout. Through the attention weight model, the model can independently decide in which cases it should pay more attention to local details and in which cases it needs a stronger global structural understanding. Next, the first image feature and the second image feature are dimensionally aligned. Dimension alignment is to ensure that two different features can be fused and calculated in the same scale and space. This step can be completed by linear transformation, projection or convolution operation. After completing the dimensional alignment, the two features are weighted and summed using the attention weight, and the attention weight model will give priority to more important features when combining these features. Finally, a comprehensive first combined image feature is obtained by weighted summation, which not only contains the local and global information of the image, but is also optimized to highlight the key areas and key information in the image.

[0108] Through the above steps, by introducing the attention mechanism, the model's attention to different image features can be adaptively adjusted, making the fusion of local details and global structures more reasonable. The combined image features after weighted summation have both fineness and global consistency, which greatly improves the visual accuracy and semantic expression ability of the model in the generation task.

[0109] Furthermore, the step of dimensionally aligning the first image feature and the second image feature specifically includes:

[0110] Generate adaptive scaling factors based on the needs of the video generation task;

[0111] Add adaptive scaling factors to the attention weight model;

[0112] Based on the adaptive scaling factor, the first image feature and the second image feature are dimensionally aligned in the attention weight model.

[0113] In this embodiment, first, an adaptive scaling factor is generated according to the specific needs of the current video generation task. For example, for scenes that require refined processing, the adaptive scaling factor may amplify the weight of local detail features, while for scenes that require global consistency, the overall features may be emphasized. Next, this adaptive scaling factor is added to the attention weight model, and the model performs further feature processing according to the guidance of the scaling factor. This scaling factor controls the model's allocation of importance to different features to a certain extent, making the model more flexible in processing local and global information. After introducing the adaptive scaling factor, the model aligns the first image feature and the second image feature in dimension. Dimension alignment is to ensure that features of two different scales can be compared and fused in the same space. By adjusting the scaling factor, the model can better match the relationship between features, especially when processing complex scenes, and can adaptively adjust the weight distribution of features.

[0114] Through the above steps, by introducing an adaptive scaling factor, the flexibility and adaptability of the attention model are significantly improved, and the weight distribution of local and global features can be adaptively adjusted according to specific task requirements.

[0115] Furthermore, based on the first combined image feature, the step of using the pre-trained first video generation model to generate an initial video corresponding to the image to be processed specifically includes:

[0116] Using a first video generation model, add video noise to the first combined image feature to obtain a first noisy video, wherein the video noise is Gaussian noise;

[0117] Using a preset denoising function to denoise the first noisy video to obtain an initial video key frame;

[0118] The initial video key frames are combined to obtain the initial video corresponding to the image to be processed.

[0119] In this embodiment, the first video generation model is first used to add video noise to the first combined image feature. The noise type is Gaussian noise. This process is part of the video generation diffusion model. By adding noise to randomly perturb the image features, the model is guided to generate new video content. The introduction of Gaussian noise gives the generation process a certain degree of randomness and diversity, which can avoid overly rigid generation results and increase the diversity and naturalness of the model generation results. After noise processing, an initial video clip containing noise is obtained, that is, the first noise video. Next, the first noise video is processed using a preset denoising function. The denoising function can remove noise through learning while retaining useful feature information and restoring the structure and details of the video. During the denoising process, the model will gradually clarify each frame in the video to ensure that it is visually consistent and coherent. The result after denoising is the key frames of the initial video. These key frames are the basic frames of the video, which provide the basic content and structure of the video and ensure that the subsequent video generation process has a reliable starting point. Finally, by combining these key frames, an initial video corresponding to the image to be processed is generated. The combination of key frames can be completed based on various factors such as time sequence and inter-frame correlation to ensure the smoothness and coherence of the video.

[0120] Through the above steps, by adding noise to the image features and using the denoising function to gradually generate video key frames, it is ensured that the generated video has sufficient diversity while maintaining a high degree of detail restoration and structural integrity. Ultimately, the generated initial video can not only capture the core content of the image to be processed.

[0121] Furthermore, the step of inputting the initial video and text encoding features into the pre-trained second video generation model to obtain the target video corresponding to the image to be processed specifically includes:

[0122] Obtaining an initial video key frame, and performing feature extraction on the initial video key frame to obtain key frame features;

[0123] Combining the key frame feature and the text encoding feature to obtain a second combined image feature;

[0124] Performing an upsampling operation on the second combined image feature, and converting the resolution of the upsampled feature into a target resolution, wherein the target resolution is a required resolution of the video generation task;

[0125] Adding video noise to the upsampled features after the resolution conversion to obtain a second noisy video;

[0126] Use a denoising function to denoise the second noisy video to obtain a target video key frame;

[0127] Combine the target video key frames to obtain the target video corresponding to the image to be processed.

[0128] In this embodiment, first, the key frames of the initial video are obtained, and these key frames contain the main content of the video. By extracting features from these key frames, key frame features can be obtained, and the key frame features reflect the structure, texture and semantic information of each frame image to a certain extent. Then, these key frame features are combined with the text encoding features obtained by text encoding to generate a second combined image feature. The addition of the text encoding feature can make the video generation process more in line with the semantic requirements of the input text, so that the video content is not only visually reasonable, but also can express the semantic information related to the text. Then, the second combined image feature is upsampled, and upsampling can improve the resolution of the feature, making the generated video more refined. In order to ensure that the output of the video meets the specific requirements, the upsampled features are also converted to a resolution to ensure that their resolution reaches the target resolution required by the video generation task. After that, video noise is added to these upsampled features again to generate a second noisy video. The addition of video noise is similar to the generation process of the diffusion model, which can increase the randomness and diversity of the generation. Finally, the second noisy video is denoised using a denoising function, and this process gradually restores the details of the video to generate the key frames of the target video. These key frames are then combined and processed to generate a complete high-resolution target video.

[0129] Through the above steps, by combining the initial video key frames with the text encoding features, and through upsampling, resolution conversion and denoising steps, a high-resolution video that meets the task requirements is generated, ensuring that the video content maintains the original information. At the same time, through multiple feature enhancement and denoising operations, the video can present clearer and more delicate details while taking into account semantic accuracy.

[0130] In the above embodiment, the present application discloses a video generation method, which belongs to the field of artificial intelligence technology. The present application obtains a first image feature by inputting the image to be processed into a pre-trained image processing model; inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a first combined image feature; based on the first combined image feature, using the pre-trained first video generation model to generate an initial video corresponding to the image to be processed; obtaining a video text description, and encoding the video text description to obtain a text encoding feature; inputting the initial video and the text encoding feature into the pre-trained second video generation model to obtain a target video corresponding to the image to be processed. The present application extracts the first image feature through the image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through the global encoder, and the second image feature captures the overall features of the image. The feature fusion of the two ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video.

[0131] In this embodiment, the video generation method is executed on the electronic device (eg Figure 1 The server shown in the figure can receive instructions or obtain data through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (Ultra Wi-Fi Band) connection, and other wireless connection methods currently known or to be developed in the future.

[0132] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned image information to be processed, the above-mentioned image information to be processed can also be stored in a node of a blockchain.

[0133] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0134] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0135] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0137] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0138] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a video generation device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0139] like Figure 3 As shown, the video generating device 300 described in this embodiment includes:

[0140] A first feature extraction module 301 is used to obtain an image to be processed and input the image to be processed into a pre-trained image processing model to obtain a first image feature;

[0141] A second feature extraction module 302, used to input the image to be processed into a pre-trained global encoder to obtain a second image feature;

[0142] A feature weight combination module 303 is used to combine the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature;

[0143] A first video generation module 304 is used to generate an initial video corresponding to the image to be processed based on the first combined image feature using a pre-trained first video generation model, wherein the first video generation model is a pre-trained video generation diffusion model;

[0144] The third feature extraction module 305 is used to obtain the video text description and encode the video text description to obtain text encoding features;

[0145] The second video generation module 306 is used to input the initial video and text encoding features into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

[0146] Furthermore, the image processing model is used to capture local features of the image. The image processing model is built based on the CLIP model architecture. The CLIP model architecture includes an image encoder and a text encoder. The first feature extraction module 301 is specifically used for:

[0147] Perform text extraction on the processed image to obtain image text; use an image encoder to encode the processed image to obtain image coding features; use a text encoder to encode the image text to obtain image text coding features; map the image coding features and the image text coding features to a preset first feature space, and perform feature alignment on the image coding features and the image text coding features in the first feature space; obtain the feature alignment result of the first feature space to obtain a first image feature.

[0148] Furthermore, the second feature extraction module 302 is specifically used for:

[0149] The image to be processed is divided into several image blocks; each image block is encoded in turn using a global encoder to obtain image block features; each image block feature is mapped in turn to a preset second feature space, and the feature correlation relationship between the features of each image block is captured in the second feature space to obtain a second image feature.

[0150] Furthermore, the feature weight combination module 303 is specifically used for:

[0151] The first image feature and the second image feature are input into a pre-trained attention weight model, and an attention weight value is output; the first image feature and the second image feature are dimensionally aligned; and the attention weight value is used to perform weighted summation on the first image feature and the second image feature after the dimension alignment to obtain a first combined image feature.

[0152] Furthermore, the feature weight combination module 303 is further used for:

[0153] An adaptive scaling factor is generated according to the requirements of the video generation task; the adaptive scaling factor is added to the attention weight model; and based on the adaptive scaling factor, the first image feature and the second image feature are dimensionally aligned in the attention weight model.

[0154] Furthermore, the first video generating module 304 is specifically used for:

[0155] Using a first video generation model, add video noise to the first combined image feature to obtain a first noisy video, wherein the video noise is Gaussian noise;

[0156] Using a preset denoising function to denoise the first noisy video to obtain an initial video key frame;

[0157] The initial video key frames are combined to obtain the initial video corresponding to the image to be processed.

[0158] Furthermore, the second video generating module 306 is specifically used for:

[0159] Obtaining an initial video key frame, and performing feature extraction on the initial video key frame to obtain key frame features;

[0160] Combining the key frame feature and the text encoding feature to obtain a second combined image feature;

[0161] Performing an upsampling operation on the second combined image feature, and converting the resolution of the upsampled feature into a target resolution, wherein the target resolution is a required resolution of the video generation task;

[0162] Adding video noise to the upsampled features after the resolution conversion to obtain a second noisy video;

[0163] Use a denoising function to denoise the second noisy video to obtain a target video key frame;

[0164] Combine the target video key frames to obtain the target video corresponding to the image to be processed.

[0165] In the above embodiment, the present application discloses a video generation device, which belongs to the field of artificial intelligence technology. The present application obtains a first image feature by inputting the image to be processed into a pre-trained image processing model; inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a first combined image feature; based on the first combined image feature, using the pre-trained first video generation model to generate an initial video corresponding to the image to be processed; obtaining a video text description, and encoding the video text description to obtain a text encoding feature; inputting the initial video and the text encoding feature into the pre-trained second video generation model to obtain a target video corresponding to the image to be processed. The present application extracts the first image feature through the image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through the global encoder, and the second image feature captures the overall features of the image. The feature fusion of the two ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video.

[0166] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0167] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0168] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0169] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the video generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0170] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions or process data stored in the memory 41, such as the computer-readable instructions for running the video generation method.

[0171] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0172] In the above embodiment, the present application discloses a computer device, which belongs to the field of artificial intelligence technology. The present application obtains a first image feature by inputting the image to be processed into a pre-trained image processing model; inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a first combined image feature; based on the first combined image feature, using a pre-trained first video generation model to generate an initial video corresponding to the image to be processed; obtaining a video text description, and encoding the video text description to obtain a text encoding feature; inputting the initial video and the text encoding feature into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed. The present application extracts the first image feature through an image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through a global encoder, and the second image feature captures the overall features of the image. The feature fusion of the two ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video.

[0173] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the video generation method as described above.

[0174] In the above embodiment, the present application discloses a computer-readable storage medium, which belongs to the field of artificial intelligence technology. The present application obtains a first image feature by inputting the image to be processed into a pre-trained image processing model; inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; combining the first image feature and the second image feature to obtain a first combined image feature; based on the first combined image feature, using a pre-trained first video generation model to generate an initial video corresponding to the image to be processed; obtaining a video text description, and encoding the video text description to obtain a text encoding feature; inputting the initial video and the text encoding feature into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed. The present application extracts the first image feature through an image processing model, and the first image feature focuses on the precise perception of image details. At the same time, the second image feature is extracted through a global encoder, and the second image feature captures the overall features of the image. The fusion of the two features ensures that the model can reproduce subtle visual details and maintain global consistency during the generation process. Finally, the diffusion model is used to generate the initial video, and high-resolution optimization is performed in combination with text encoding, which effectively improves the visual quality and semantic consistency of the video.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0176] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0177] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A video generation method, characterized in that: include: Acquire an image to be processed, and input the image to be processed into a pre-trained image processing model to obtain a first image feature; Inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; Combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature; Based on the first combined image feature, using a pre-trained first video generation model to generate an initial video corresponding to the image to be processed, wherein the first video generation model is a pre-trained video generation diffusion model; Obtaining a video text description, and encoding the video text description to obtain a text encoding feature; The initial video and the text encoding features are input into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

2. The video generation method according to claim 1, characterized in that: The image processing model is used to capture local features of an image. The image processing model is built based on a CLIP model architecture. The CLIP model architecture includes an image encoder and a text encoder. The step of obtaining an image to be processed and inputting the image to be processed into a pre-trained image processing model to obtain a first image feature specifically includes: Performing text extraction on the image to be processed to obtain image text; Encoding the image to be processed using the image encoder to obtain image coding features; Encode the image text using the text encoder to obtain image text encoding features; Mapping the image coding features and the image text coding features to a preset first feature space, and performing feature alignment on the image coding features and the image text coding features in the first feature space; Acquire the feature alignment result of the first feature space to obtain the first image feature.

3. The video generation method according to claim 1, characterized in that: The step of inputting the image to be processed into a pre-trained global encoder to obtain a second image feature specifically includes: Performing image division on the image to be processed to obtain a plurality of image blocks; Using the global encoder to encode each of the image blocks in turn to obtain image block features; Each of the image block features is sequentially mapped to a preset second feature space, and a feature association relationship between each of the image block features is captured in the second feature space to obtain the second image feature.

4. The video generation method according to claim 1, characterized in that: The step of combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature specifically includes: Inputting the first image feature and the second image feature into a pre-trained attention weight model, and outputting an attention weight value; Dimensionally aligning the first image feature and the second image feature; The attention weight value is used to perform weighted summation on the first image feature and the second image feature after the dimension alignment is completed to obtain the first combined image feature.

5. The video generation method according to claim 4, characterized in that: The step of dimensionally aligning the first image feature and the second image feature specifically includes: Generate adaptive scaling factors based on the needs of the video generation task; adding the adaptive scaling factor to the attention weight model; Based on the adaptive scaling factor, dimensionally aligning the first image feature and the second image feature in the attention weight model.

6. The video generation method according to claim 1, characterized in that: The step of generating an initial video corresponding to the image to be processed using a pre-trained first video generation model based on the first combined image feature specifically includes: Using the first video generation model, adding video noise to the first combined image feature to obtain a first noisy video, wherein the video noise is Gaussian noise; Using a preset denoising function to denoise the first noisy video to obtain an initial video key frame; The initial video key frames are combined to obtain an initial video corresponding to the image to be processed.

7. The video generation method according to claim 6, characterized in that: The step of inputting the initial video and the text encoding features into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed specifically includes: Acquire the initial video key frame, and perform feature extraction on the initial video key frame to obtain key frame features; Combining the key frame feature and the text encoding feature to obtain a second combined image feature; Performing an upsampling operation on the second combined image feature, and converting the resolution of the upsampled feature to a target resolution, wherein the target resolution is a required resolution of a video generation task; Adding video noise to the upsampled features after the resolution conversion to obtain a second noisy video; Performing denoising processing on the second noisy video using the denoising function to obtain a target video key frame; The target video key frames are combined to obtain a target video corresponding to the image to be processed.

8. A video generating device, characterized in that: include: A first feature extraction module is used to obtain an image to be processed and input the image to be processed into a pre-trained image processing model to obtain a first image feature; A second feature extraction module, used for inputting the image to be processed into a pre-trained global encoder to obtain a second image feature; A feature weight combination module, used for combining the first image feature and the second image feature based on a preset feature weight algorithm to obtain a first combined image feature; A first video generation module, configured to generate an initial video corresponding to the image to be processed using a pre-trained first video generation model based on the first combined image feature, wherein the first video generation model is a pre-trained video generation diffusion model; A third feature extraction module is used to obtain a video text description and encode the video text description to obtain a text encoding feature; The second video generation module is used to input the initial video and the text encoding features into a pre-trained second video generation model to obtain a target video corresponding to the image to be processed, wherein the second video generation model is a pre-trained video high-resolution model.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the video generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the video generation method according to any one of claims 1 to 7 are implemented.

Images