Data processing method and device

Abstract

The present application discloses a data processing method and device, which relates to the field of computer technology. The method includes: obtaining a target text, obtaining P consecutive frames of text from the target text; obtaining P noise images, obtaining the target frame text from the P consecutive frames of text, and determining the target noise image from the P noise images; processing the target noise image using a target video generation model to obtain a target forward noise vector; processing the business text, business image, and target frame text using the target video generation model to obtain a fused coding feature, denoising the target noise image based on the target forward noise vector and the fused coding feature, obtaining a target predicted image corresponding to the target frame text, and using the target predicted image as the next frame image located in the business image in the video to be generated. By using this application, the overall tone consistency of the Wensheng video can be improved, and the quality of the Wensheng video can be guaranteed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of artificial intelligence (AI), particularly the continuous advancements in generative adversarial networks and diffusion models, significant breakthroughs have been achieved in the field of image and video generation. Text-to-video technology can generate images based solely on textual descriptions. The core of this technology lies in combining natural language processing with computer vision techniques, leveraging trained models to transform textual information into visual content. Building on this foundation of image generation technology, text-to-video (Text-to-Video) further expands upon this capability. Text-to-video requires models to not only generate static images but also generate continuous dynamic images based on textual descriptions. Currently, using the open-source stable diffusion model (SD model), text-to-video can be used to generate the first frame of an image using specific control parameters. Then, using the same control parameters, the second frame is generated until all the images required for the video sequence are generated, resulting in a video. However, when the generated text input to the SD model changes relative to the previous frame, the generated foreground and background of the next frame can be significantly inconsistent with the previous frame, resulting in poor generation quality and low-quality text-to-video. Summary of the Invention

[0003] The embodiments of the present application provide a data processing method and device, which can improve the overall tone consistency of Vincent videos and ensure the quality of Vincent videos.

[0004] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0005] Obtain the target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer;

[0006] Obtain P noise images, obtain target frame texts traversed in order from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images;

[0007] The target noise image is input into the target video generation model, and the target noise image is subjected to feature extraction by the target video generation model to obtain target noise features, and the target noise features are subjected to forward diffusion processing to obtain a target forward noise vector;

[0008] The business text, business image, and target frame text are input into the target video generation model. The target frame text is encoded by the target video generation model to obtain text encoding features. Multimodal feature extraction is performed on the business text and business image to obtain image-text encoding features. The business image belongs to the frame image generated in the video to be generated. The business text is the continuous frame text used to represent the business image among P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0009] The target noisy image is denoised according to the target forward noise vector, text encoding features and image-text encoding features to obtain the target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located in the business image in the video to be generated.

[0010] On the one hand, an embodiment of the present application provides another data processing method, which includes:

[0011] Obtain a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0012] Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples;

[0013] Extract features of the target image sample and the target noise sample corresponding to the target image sample in N noise samples to obtain potential noise features, perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector;

[0014] Performing text encoding on the target text sample to obtain a first encoding feature, and performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample;

[0015] Performing back diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector;

[0016] According to N noise samples and predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0017] In one aspect, an embodiment of the present application provides a data processing device, the device comprising:

[0018] The data acquisition module is used to acquire the target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer;

[0019] A data determination module is used to obtain P noise images, obtain the target frame text traversed in order from the P consecutive frame texts, and determine the target noise image associated with the target frame text from the P noise images;

[0020] A data forward diffusion module is used to input the target noise image into the target video generation model, extract the features of the target noise image through the target video generation model to obtain the target noise features, and perform forward diffusion processing on the target noise features to obtain the target forward noise vector;

[0021] A feature encoding module is used to input the business text, business image, and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, and perform multimodal feature extraction on the business text and business image to obtain image-text encoding features; the business image is a frame image that has been generated in the video to be generated; the business text is a continuous frame text used to represent the business image among P continuous frame texts, and the business text is a continuous frame text that is sorted before the target frame text;

[0022] The image prediction module is used to denoise the target noisy image based on the target forward noise vector, text encoding features and image-text encoding features, obtain the target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located in the business image in the video to be generated.

[0023] In a possible implementation, when the data determination module is used to obtain P noisy images, the data determination module is specifically used to perform the following operations:

[0024] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0025] Randomly obtain M noise data from a noise data pool, splice the M noise data to obtain a first spliced ​​noise map, perform Fourier transform on the first spliced ​​noise map to obtain a first frequency domain corresponding to the first spliced ​​noise map, and determine a frequency signal in the first frequency domain that is less than a low-frequency threshold as a global low-frequency signal; M is a positive integer less than S;

[0026] Obtain a first sliding step length, obtain P noise data groups from the S noise data using the first sliding step length, and splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M;

[0027] Performing Fourier transform on each of the P second spliced ​​noise images to obtain second frequency domains corresponding to the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to a low-frequency threshold as high-frequency signals;

[0028] The high-frequency signal in each second frequency domain is combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are inverse Fourier transformed to obtain P noise images.

[0029] In a possible implementation, when the data determination module is used to obtain P noisy images, the data determination module is further used to perform the following operations:

[0030] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0031] Obtain a second sliding step length, obtain P noise data groups from the S noise data through the second sliding step length, and splice the noise data in the P noise data groups to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step length is less than M.

[0032] In one possible implementation, the feature encoding module is used to extract multimodal features from the business text and the business image. When obtaining the image-text encoding features, the feature encoding module is specifically used to perform the following operations:

[0033] Through the image encoding component in the target video generation model, feature extraction is performed on the business image to obtain image extraction features;

[0034] Through the text encoding component in the target video generation model, feature extraction is performed on the business text to obtain text extraction features;

[0035] The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are aligned to obtain the image-text encoding features.

[0036] In one possible implementation, the image prediction module is configured to perform denoising on the target noisy image based on the target forward noise vector, the text encoding features, and the image-text encoding features. When obtaining the target predicted image corresponding to the target frame text, the image prediction module is specifically configured to perform the following operations:

[0037] Generate fusion coding features based on text coding features and image-text coding features;

[0038] In the back diffusion network layer of the target video generation model, based on the fusion coding features, the target forward noise vector is back-diffused in T time steps to obtain the target prediction noise; T is a positive integer;

[0039] Determine the target latent vector based on the target prediction noise and the target forward noise vector;

[0040] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.

[0041] In one possible implementation, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; when the image prediction module is used to generate a fused coding feature based on the text coding feature and the image-text coding feature, the image prediction module is specifically used to perform the following operations:

[0042] Perform feature splicing on the text coding features and the image-text coding features to obtain spliced ​​coding features;

[0043] The concatenated coding features are dimensionally fused through the feature fusion network in the target video generation model to obtain fused coding features; the fused coding features have the same dimension as the target forward noise vector.

[0044] In a possible implementation, the number of business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the to-be-generated video; the image-text coding feature includes F image-text coding sub-features, and each image-text coding sub-feature is obtained by performing multimodal feature extraction based on a business text and the represented business image;

[0045] When the image prediction module is used to generate fused coding features based on text coding features and image-text coding features, the image prediction module is specifically used to perform the following operations:

[0046] The unit features in the F image-text encoding sub-features are combined into a sequence to be matched;

[0047] Perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature, sort each unit feature based on the feature matching degree of each unit feature, and obtain a feature matching sequence;

[0048] The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer;

[0049] The text encoding features and target matching features are concatenated to obtain fused encoding features.

[0050] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module is used to perform back diffusion processing on the target forward noise vector in T time steps based on the fused coding features. When the target prediction noise is obtained, the image prediction module is specifically used to perform the following operations:

[0051] At time step T i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;

[0052] The time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.

[0053] In one possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network and a bypass attention sub-network; the image prediction module is used to i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is obtained, the image prediction module is used to perform the following operations:

[0054] At time step T i In the fusion encoding feature and iterative noise vector, input to the time step T i The corresponding noise prediction network;

[0055] By time step T i The corresponding cross-attention sub-network in the noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the first attention processing result;

[0056] By time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the second attention processing result;

[0057] Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

[0058] In one possible implementation, the image prediction module is configured to perform a first cross-attention process on the fused coding feature and the initial noise vector. When the first attention process result is obtained, the image prediction module is specifically configured to perform the following operations:

[0059] The initial noise vector is used as the query vector in the crisscross attention function, the fused encoded features are used as the key vector in the crisscross attention function, and the fused encoded features are used as the value vector in the crisscross attention function;

[0060] The product of the query vector and the key vector is determined as the first fusion feature through the cross attention function;

[0061] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;

[0062] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the first attention processing result.

[0063] In a possible implementation, the data processing device further includes a video determination module, which is specifically configured to perform the following operations:

[0064] When the video to be generated contains P consecutive frame images corresponding to the text, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include the target prediction image;

[0065] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.

[0066] In a possible implementation, the frame images corresponding to the P consecutive frames of text in the video to be generated include frame images P j and frame image Pj+1 , j is a positive integer less than P; the video determination module is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module is specifically used to perform the following operations:

[0067] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;

[0068] Based on the frame image P j , frame image P j+1 And the optical flow prediction results are used to generate the frame image P j and frame image P j+1 The intermediate frame images between.

[0069] In one aspect, an embodiment of the present application provides another data processing device, the device comprising:

[0070] The sample acquisition module is used to acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0071] A sample determination module is used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples;

[0072] The sample forward diffusion module is used to extract features of the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain potential noise features, and perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector;

[0073] a sample encoding module, configured to perform text encoding on a target text sample to obtain a first encoding feature, and perform multimodal feature extraction on a first image sample and a first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample;

[0074] A sample back diffusion module is used to perform back diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector;

[0075] The parameter adjustment module is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0076] In a possible implementation, when the sample acquisition module is used to acquire a sample sequence, the sample acquisition module is specifically used to perform the following operations:

[0077] Obtain a sample video, obtain N video frames based on the order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same subject object and object scene, and the display perspectives of the subject object in the N image samples are different from each other;

[0078] Generating text samples for N image samples respectively to obtain N text samples; the text samples are used to indicate specific content in the corresponding image samples;

[0079] Obtain N noise samples, combine the N image samples, N text samples, and N noise samples to obtain a sample sequence.

[0080] In a possible implementation, when the sample acquisition module is used to acquire N noise samples, the sample acquisition module is specifically used to perform the following operations:

[0081] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0082] Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine the frequency signal in the first sample frequency domain that is less than the sample low-frequency threshold as the sample global low-frequency signal; R is a positive integer less than S;

[0083] Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data using the first sample sliding step length, and concatenate the noise data in the N sample noise data groups to obtain N second sample noise maps; the number of noise data included in each sample noise data group is R;

[0084] Performing Fourier transform on each of the N second sample noise images to obtain second sample frequency domains corresponding to the N second sample noise images, and determining frequency signals in the N second sample frequency domains that are greater than or equal to a sample low-frequency threshold as sample high-frequency signals;

[0085] Each sample high-frequency signal in the second sample frequency domain is combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are inverse Fourier transformed to obtain N noise samples.

[0086] In a possible implementation, when the sample acquisition module is used to acquire N noise samples, the sample acquisition module is further used to perform the following operations:

[0087] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0088] Obtain a second sample sliding step length, obtain N noise data groups from the S noise data through the second sample sliding step length, and splice the noise data in the N noise data groups to obtain N noise samples; the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step length is less than R.

[0089] In one possible implementation, the initial video generation model includes an initial back diffusion network layer, which includes a cross-attention subnetwork and an initial bypass attention subnetwork. The sample back diffusion module is used to perform back diffusion processing on the sample forward noise vector, the first encoding feature, and the second encoding feature. When the predicted noise sample vector is obtained, the sample back diffusion module is specifically used to perform the following operations:

[0090] generating a target coding feature based on the first coding feature and the second coding feature;

[0091] Through the cross-attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the first sample attention result;

[0092] Through the initial bypass attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the second sample attention result;

[0093] Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

[0094] In one possible implementation, the initial video generation model further includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature. The initial feature fusion network is used to generate a target coding feature based on the first coding feature and the second coding feature. The parameter adjustment module is used to adjust the parameters of the initial video generation model based on N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples. When the target video generation model is obtained, the parameter adjustment module is specifically used to perform the following operations:

[0095] Generate a model loss value based on N noise samples and the predicted noise sample vectors predicted by the N noise samples;

[0096] Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain the target video generation model.

[0097] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;

[0098] The processor is connected to the memory and the input and output interface respectively, wherein the input and output interface is used to receive and output data, the memory is used to store the computer program, and the processor is used to call the computer program so that the computer device including the processor executes the method in one aspect of the embodiment of the present application.

[0099] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method in one aspect of the embodiment of the present application.

[0100] In one aspect, an embodiment of the present application provides a computer program product, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional embodiments of the present application. In other words, when the computer program is executed by the processor, the methods provided in various optional embodiments of the present application are implemented.

[0101] In an embodiment of the present application, a target text is obtained, the target text is split and processed to obtain P consecutive frame texts in a video frame sorting order; P is a positive integer. P noise images are obtained, the target frame text traversed in order is obtained from the P consecutive frame texts, and the target noise image associated with the target frame text is determined from the P noise images. The target noise image is input into a target video generation model, and the target noise image is subjected to feature extraction by the target video generation model to obtain target noise features, and the target noise features are subjected to forward diffusion processing to obtain a target forward noise vector. Business text, business image and target frame text are input into a target video generation model, and the target frame text is subjected to text encoding by the target video generation model to obtain text encoding features, and multimodal feature extraction is performed on the business text and business image to obtain graphic encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text used to characterize the business image in the P continuous frame texts, and the business text is a continuous frame text sorted before the target frame text. The feature extraction method can be fully utilized to extract the features of historical images (business images) and historical texts (business texts); the target noise image can be denoised according to the target forward noise vector, text encoding features and image-text encoding features to obtain the target predicted image corresponding to the target frame text, and the target predicted image can be used as the next frame image located in the business image in the video to be generated. The business text and business image are combined with the latest frame text (target frame text) to jointly constrain and guide the target video generation model, and the target prediction image corresponding to the target frame text is generated through the target noise image, ensuring that the content (main object and object scene, etc.) of the generated target prediction image and the historical image (i.e., the latest frame in the video to be generated, i.e., the target prediction image and the historical frame) can be consistent, avoiding the excessive difference between the generated latest frame and the historical frame due to the change of the latest frame text, improving the similarity between the target prediction image and the historical image, and thus improving the coherence and coordination of the video to be generated based on the target text; at the same time, the constraints of historical text and historical image on the generation of target prediction image by the target video generation model can prompt the generation style or content of the target prediction image to be biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby improving the overall tone consistency of the cultural video and further ensuring the quality of the cultural video. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0103] Figure 1 This is a network interaction architecture diagram provided by an embodiment of the present application;

[0104] Figure 2 This is a scenario diagram of a data processing method provided in an embodiment of the present application;

[0105] Figure 3 This is a data processing method provided by the embodiment of the present application. Figure 1 ;

[0106] Figure 4 This is a data processing method provided by the embodiment of the present application. Figure 2 ;

[0107] Figure 5 This is a noise image acquisition scenario provided by the embodiment of the present application. Figure 1 ;

[0108] Figure 6 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 1 ;

[0109] Figure 7 This is a schematic diagram of a downsampling network layer structure provided in an embodiment of the present application;

[0110] Figure 8 This is a schematic diagram of a transformation processing layer structure provided by an embodiment of the present application;

[0111] Figure 9 This is a schematic diagram of the structure of a transformation processing unit provided in an embodiment of the present application;

[0112] Figure 10 This is a data processing method provided by the embodiment of the present application. Figure 3 ;

[0113] Figure 11 This is a noise image acquisition scenario provided by the embodiment of the present application. Figure 2 ;

[0114] Figure 12 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 2 ;

[0115] Figure 13 This is a data processing method provided by the embodiment of the present application. Figure 4 ;

[0116] Figure 14 This is a schematic diagram of the structure of an initial video generation model provided in the embodiment of the present application. Figure 1 ;

[0117] Figure 15This is a data processing method provided by the embodiment of the present application. Figure 5 ;

[0118] Figure 16 This is a schematic diagram of the structure of an initial video generation model provided in the embodiment of the present application. Figure 2 ;

[0119] Figure 17 This is a schematic diagram of a data processing device provided in an embodiment of the present application. Figure 1 ;

[0120] Figure 18 This is a schematic diagram of a data processing device provided in an embodiment of the present application. Figure 2 ;

[0121] Figure 19 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0122] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0123] Among them, if it is necessary to collect object (such as user, etc.) data in this application, a prompt interface or pop-up window will be displayed before or during the collection. The prompt interface or pop-up window is used to remind the user that certain data is currently being collected. Only after the user confirms the prompt interface or pop-up window, the relevant steps for data acquisition will be started, otherwise the process will end. Moreover, the acquired user data will be used in reasonable and legal scenarios or purposes. Optionally, in some scenarios where user data needs to be used but has not been authorized by the user, authorization can be requested from the user, and the user data can be used when the authorization is passed.

[0124] It is understandable that in the specific implementation of this application, the user data involved, when the following embodiments of this application are applied to specific products or technologies, needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.

[0125] In the examples of this application, see Figure 1 , Figure 1 This is a network interaction architecture diagram provided by an embodiment of the present application, such as Figure 1As shown, the network interaction architecture diagram may include a business server 101 and a terminal device cluster, and the terminal device cluster may include terminal device 102a, terminal device 102b, terminal device 102c, ..., terminal device 102n, wherein a communication connection may exist between the terminal device clusters, for example, a communication connection exists between terminal device 102a and terminal device 102b, and a communication connection exists between terminal device 102a and terminal device 102c. At the same time, any terminal device in the terminal device cluster may have a communication connection with the business server 101, for example, a communication connection exists between terminal device 102a and the business server 101, wherein the above-mentioned communication connection does not limit the connection method, and may be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and this application does not make any restrictions here.

[0126] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown can be installed with an application client with data processing function. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1 The business servers 101 shown in the figure interact with each other so that the business server 101 can receive business data from each terminal device. Among them, the application client can be a social application, instant messaging application, live broadcast application, short video application, video application, music application, shopping application, novel application, browser, etc., which has the function of displaying text, images, audio, video and other data information. Among them, the application client can be an independent client or an embedded sub-client (for example, an application applet, browser web page access, etc.) integrated in a client (for example, an instant messaging client, a social client, a video client, etc.), which is not limited here.

[0127] like Figure 1As shown, any terminal device in the terminal device cluster can send a video generation request containing target text to the business server 101. Taking terminal device 102a as an example, terminal device 102a can send the target text to the business server 101 via the video generation request. The target text is the descriptive text of the video to be generated indicated by the video generation request. The business server 101 can receive the video generation request sent by terminal device 102a, obtain the target text from the video generation request, expand the target text based on the semantic information of the target text, and split the expanded target text to obtain P consecutive frame texts with a video frame sorting order; P is a positive integer. The P consecutive frame texts are respectively used to generate the P frame images in the video to be generated, and are respectively used to represent the descriptive text of the P frame images in the video to be generated. It can be understood that in order to maintain the coherence of the video to be generated, the difference between two adjacent consecutive frame texts in the P consecutive frame texts is small, that is, the text description content or style of the two consecutive frame texts are similar. For example, the target text can be represented as "A cat chases butterflies on the grass, then suddenly jumps onto a stone, and finally sits on the stone and licks its paws." The texts of two adjacent consecutive frames can be represented as "The cat starts to lick its front paw with its tongue" and "The cat switches to the other paw and continues to lick it with its tongue" respectively.

[0128] The business server 101 can obtain P noise images, determine the target frame text from the P consecutive frame texts by traversing in order, and determine the target noise image associated with the target frame text from the P noise images. The business server 101 can extract features from the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector. The business server 101 can input the business text, business image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image to obtain image-text encoding features. Among them, the business image belongs to the frame image that has been generated in the video to be generated, that is, the historical frame image; the business text is the continuous frame text in the P continuous frame texts used to represent the business image, and the business text is the continuous frame text sorted before the target frame text.

[0129] The service server 101 can perform denoising on the target noise image based on the target forward noise vector, text encoding features, and image-text encoding features to obtain a target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located in the service image in the video to be generated. When the target predicted image corresponding to the last continuous frame text is generated, the service video indicated by the target text can be obtained. The service video includes frame images corresponding to P continuous frame texts, i.e., P frame images. The service server 101 can send the service video to the terminal device 102a so that the terminal device 102a can display the service video.

[0130] The embodiment of the present application jointly constrains and guides the target video generation model through business images, business texts, and target frame texts to generate a target prediction image for the target frame text, so that the features of the business images and business texts can play a role in the process of generating the target prediction image, ensuring that the generated target prediction image and the content (subject object and object scene, etc.) of the historical image (i.e., the newly generated frame and historical frame of the video to be generated) can be consistent, avoiding excessive differences in the generated images due to changes in the latest frame text (target frame text), improving the similarity between the target prediction image and the historical image, and thus improving the coherence and coordination of the video to be generated based on the target text; at the same time, the constraints of historical texts and historical images on the generation of target prediction images by the target video generation model can prompt the generation style or content of the target prediction image to be biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby improving the overall tone consistency of the Wensheng video and further ensuring the quality of the Wensheng video.

[0131] It is understandable that the terminal device mentioned in the embodiments of the present application may also be a computer device, and the computer device in the embodiments of the present application includes but is not limited to a terminal device or a server. In other words, the computer device may be a server or a terminal device, or it may be a system composed of a server and a terminal device. Among them, the terminal device mentioned above may be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. As Figure 1As shown in , the terminal device may be a mobile phone (as shown by terminal device 102a), a desktop computer (as shown by terminal device 102b), a tablet computer (as shown by terminal device 102c) or a laptop computer (as shown by terminal device 102n), etc. Figure 1 Only some of the devices are listed. The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0132] Optionally, the data involved in the embodiments of the present application can be stored in a computer device, or the data can be stored based on cloud storage technology or a blockchain network, which is not limited here.

[0133] For details, see Figure 2 , Figure 2 This is a scenario diagram of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the service server can obtain the target text in the video generation request sent by the terminal device, wherein the service server can be Figure 1 The service server 101 shown, the terminal device can be Figure 1 Any terminal device in the terminal device cluster shown, here taking terminal device 102a as an example. The user corresponding to terminal device 102a inputs a video description (target text) and initiates a video generation request. The service server 101 can obtain the target text in the video generation request sent by terminal device 102a, split the target text into P continuous frame texts (including continuous frame text C1, continuous frame text C2, ..., continuous frame text C P ). The service server 101 can obtain P noise images, obtain the target frame text traversed in order from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images. For example, when the target frame text is the first continuous frame text (continuous frame text C1) among the P continuous frame texts, the target noise image can be the first noise image among the P noise images. Similarly, when the target frame text is the last continuous frame text (continuous frame text C1) among the P continuous frame texts, the target noise image can be the first noise image among the P noise images. P ), the target noise image may be the last noise image among the P noise images, that is, the Pth noise image.

[0134] The service server 101 can input the target noise image into the target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector. The target video generation model can be modified based on the SD model, and the service server 101 can perform forward diffusion processing on the target noise features through the forward diffusion network layer of the target video generation model to obtain a target forward noise vector. The forward diffusion network layer can be the forward diffusion network layer in the SD model. The service server 101 can input service text, service image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features; perform multimodal feature extraction on the service text and service image to obtain image-text encoding features. Among them, the business image belongs to the frame image generated in the video to be generated, that is, the continuous frame text (historical text) sorted before the target frame text, and the target predicted image corresponding to each historical text generated by the target video generation model; the business text is the continuous frame text used to represent the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text. For example, when the target frame text is the third continuous frame text (i.e., continuous frame text C3) in P continuous frame texts, the business text can be the continuous frame text sorted before the target frame text, that is, including the first continuous frame text (i.e., continuous frame text C1) and the second continuous frame text (i.e., continuous frame text C2) in P continuous frame texts. It should be noted that when the target frame text is the first continuous frame text (i.e., continuous frame text C1) in P continuous frame texts, the business image and business text input by the business server 101 to the target video generation model can be null inputs, that is, the input business image is a blank null image, and the input business text is a blank null text.

[0135] Furthermore, the service server 101 can generate a target video model and, based on the target forward noise vector, text encoding features, and image-text encoding features, perform denoising on the target noise image to obtain a target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located at the service image in the video to be generated. When the service server 101 obtains target prediction images corresponding to P consecutive frames of text, that is, P target prediction images (including target prediction image B1, target prediction image B2, ..., target prediction image B P ), the final video to be generated can be obtained. The final video to be generated includes P frame images, the target prediction image B1 is the target prediction image corresponding to the continuous frame text C1, the target prediction image B2 is the target prediction image corresponding to the continuous frame text C2, and the target prediction image B P For continuous frame text CP The corresponding target predicted image. Optionally, the business server 101 can further perform frame insertion processing on the final video to be generated, further improve the inter-frame coherence of the video to be generated, and obtain a business video. That is, the business server 101 inserts one or more intermediate frame images into every two adjacent frame images in the P frame images of the final video to be generated, so as to make the transition between the two adjacent frame images more natural, and at the same time make the obtained business video more coherent. Among them, the business video includes the P frame images in the video to be generated, and the intermediate frame images inserted between every two adjacent frame images. The business server 101 can send the business video to the terminal device 102a, so that the terminal device 102a can display the business video to the user at the front end.

[0136] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target predicted image for the target frame text, so that the features of the business image and business text play a role in the process of generating the target predicted image, which ensures that the content (subject and object scene, etc.) of the generated target predicted image and the historical image (i.e., the newly generated frame and historical frame in the to-be-generated video) can be consistent, avoiding excessive differences in the generated image due to changes in the latest frame text (i.e., the target frame text), improving the similarity between the target predicted image and the historical image, and thus improving the coherence and coordination of the to-be-generated video generated based on the target text; at the same time, the constraints imposed by the historical text and historical image on the generation of the target predicted image by the target video generation model can encourage the target predicted image to be generated in a style or content that is biased towards the historical image, ensuring that the difference between the target predicted image and the historical image is not large, thereby improving the overall tone consistency of the generated video, and inserting an intermediate frame image between every two adjacent frame images in the to-be-generated video to further improve the coherence of the business video indicated by the target text, further ensuring the quality of the generated video.

[0137] The embodiments of the present application can generate multiple consistent target prediction images for a main target text during the video generation process, ensuring the consistency and continuity of the content of the multiple target prediction images, thereby ensuring the coherence of the business video composed of multiple target prediction images. This can provide an efficient technical means for the implementation of multiple video generation scenarios, or for subsequent business video generation given a certain initial image and target text. Potential application directions may include:

[0138] 1. In terms of social media, users can be given a travel picture, pet picture and text description (target text) to create a video of subsequent frames, thus providing users with an interesting gameplay experience.

[0139] 2. In terms of virtual display pictures, in e-commerce, merchants can upload photos of a product, and then use the product pictures and different text descriptions (such as the occasions and backgrounds in which the product appears) to freely create product displays, resulting in a variety of visual effect design results for users to choose from.

[0140] Further, see Figure 3 , Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will be described by taking the data processing method executed by a computer device as an example. Among them, the data processing method may at least include the following steps S301-S305:

[0141] Step S301: Obtain target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer.

[0142] In an embodiment of the present application, a computer device can obtain a target text, wherein the target text can come from a video generation request uploaded by a user corresponding to the computer device, or from a video generation request sent by another computer device, and the target text is used to characterize the video content description of the video to be generated in response to user needs. Furthermore, the computer device can parse the target text, obtain semantic information, expand the target text based on the semantic information, and split the expanded target text into P consecutive frame texts with a video frame sorting order; P is a positive integer. The P consecutive frame texts are respectively used to generate P frame images in the video to be generated, and are respectively used to characterize the description text of the P frame images in the video to be generated. For example, two adjacent consecutive frame texts in the P consecutive frame texts can be respectively expressed as "the cat begins to lick its front paw with its tongue" and "the cat switches to the other paw and continues to lick it with its tongue."

[0143] Step S302 , obtaining P noise images, obtaining a target frame text traversed in sequence from the P consecutive frame texts, and determining a target noise image associated with the target frame text from the P noise images.

[0144] In an embodiment of the present application, a computer device can obtain P noise images from a memory, wherein the P noise images can be determined by the computer device based on noise data obtained from a noise data pool prepared in advance. Each noise image in the P noise images has the same low-frequency signal, and the high-frequency signals between each noise image are different. The computer device obtains the target frame text that is traversed in sequence from the P consecutive frame texts, that is, the computer device performs image generation processing on the consecutive frame texts one by one in the order of the P consecutive frame texts. After determining the target frame text, the computer device can assign a corresponding target noise image to the target frame text based on the P noise images. For example, when the target frame text is the first consecutive frame text in the P consecutive frame texts, the first noise image in the P noise images is determined as the target noise image associated with the target frame text; when the target frame text is the second consecutive frame text in the P consecutive frame texts, the second noise image in the P noise images is determined as the target noise image associated with the target frame text, and so on, and the target noise image associated with the target frame text is determined in sequence from the P noise images.

[0145] Step S303: input the target noise image into the target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.

[0146] In an embodiment of the present application, a computer device may input a target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model, and obtain a mean vector and variance vector of the target noise image. The mean vector may be the average value of all pixel values ​​of the target noise image on each channel, i.e., the first-order statistic of the target noise image, and the variance vector may be the average value of the variance of the pixel values ​​of each channel. i.e., the second-order statistic of the target noise image. The computer device may randomly sample the mean vector and variance vector of the target noise image to obtain a potential mean vector and a potential variance vector, and generate a target noise feature based on the potential mean vector and the potential variance vector. Furthermore, in the forward diffusion network layer of the target video generation model, the computer device may obtain a latent variable distribution, wherein the latent variable distribution is a conceptual distribution of added noise, for example, a Gaussian distribution. The computer device may continuously add a random noise vector to the target noise feature in T time steps to obtain a target forward noise vector. The time step refers to the amplitude of adding noise to the target noise feature based on the latent variable distribution. Noise can refer to unnecessary or redundant interference information present in the image data. When T is large enough, the target forward noise vector can be used to represent an image that is completely noisy.

[0147] Step S304: input the business text, business image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image to obtain graphic encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to represent the business image among P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0148] In an embodiment of the present application, the target video generation model further includes a text encoding layer and a graphic encoding layer. The computer device can perform text segmentation on the target frame text through the text encoding layer to obtain a word-gram sequence corresponding to the target frame text. Each word-gram in the word-gram sequence corresponding to the target frame text is encoded separately to obtain text encoding features. Among them, a word-gram in the word-gram sequence refers to the smallest basic unit obtained after the target frame text is segmented. The text segmentation method can be based on word segmentation (word-based), character segmentation (character-based), or subword segmentation (subword-based), and the embodiment of the present application does not limit this. Furthermore, the computer device can perform multimodal feature extraction on the business text and business image through the graphic encoding layer to obtain graphic encoding features. It is understandable that the graphic encoding layer may include a text feature extraction component and an image feature extraction component, wherein the text feature extraction component is used to extract features of the business text, and the image feature extraction component is used to extract features of the business image. The process by which the computer device extracts features of the business text and obtains text extraction features can be the same as the process of extracting features of the target frame text through the text encoding layer and obtaining text encoding features. The computer device fuses the text extraction features and the image extraction features to obtain the image-text encoding features. The feature fusion method can be concatenation, element-wise addition, or more complex interactive fusion technology, which is not limited here. Among them, the business image is a frame image generated in the video to be generated; the business text is the continuous frame text used to represent the business image in the P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text. In other words, the business text is one or more continuous frame texts (i.e., historical text) sorted before the target frame text in the P continuous frame texts; the business image is the corresponding target predicted image (i.e., historical image) obtained by the computer device through image generation based on each business text. It should be noted that when the target frame text is the first continuous frame text in the P continuous frame texts, there is no previous continuous frame text for the target frame text, that is, there is no business text, and there is no business image. At this time, the business text and business image input by the computer device to the target video generation model can be null inputs, that is, the business text is blank null text and the business image is blank null image.

[0149] Step S305, denoising the target noisy image according to the target forward noise vector, text coding features and image-text coding features, obtaining a target predicted image corresponding to the target frame text, and using the target predicted image as the next frame image located in the business image in the video to be generated.

[0150] In an embodiment of the present application, a computer device can generate a fused coding feature based on the text coding feature and the image-text coding feature. For example, when the number of image coding features is 1, and the feature dimensions of the text coding feature and the image-text coding feature are both a, a is a positive integer. After the computer device splices the text coding feature and the image-text coding feature, it obtains a spliced ​​coding feature with a feature dimension of 2a. The computer device can perform cross-channel information interaction through a 1×1 convolutional network (Network), reduce the dimension of the spliced ​​coding feature, and obtain a fused coding feature with a feature dimension of a, that is, compress the channel feature from 2a back to a channel feature. Among them, the fused coding feature has the same dimension as the target forward noise vector. Furthermore, the computer device performs reverse diffusion processing on the fused coding feature and the target forward noise vector in the reverse diffusion network layer of the target video generation model to obtain the target prediction noise. The back-diffusion network layer may include a cross-attention sub-network and a bypass-attention sub-network. The computer device may perform cross-attention processing on the fused coding features and the target forward noise vector through the cross-attention sub-network of the back-diffusion network layer to obtain a first attention processing result; and perform cross-attention processing on the fused coding features and the target forward noise vector through the bypass-attention sub-network of the back-diffusion network layer to obtain a second attention processing result. A first cross-attention coefficient and a second cross-attention coefficient are obtained, and based on the first cross-attention coefficient and the second cross-attention coefficient, the first attention processing result and the second attention processing result are weighted to obtain a cross-attention weighted result. The target prediction noise is determined based on the cross-attention weighted result.

[0151] Optionally, the number of cross-attention subnetworks and bypass-attention subnetworks can be D, where D is a positive integer. For example, the D cross-attention subnetworks include cross-attention subnetwork 1, cross-attention subnetwork 2, ..., cross-attention subnetwork D, and the D bypass-attention subnetworks include bypass-attention subnetwork 1, bypass-attention subnetwork 2, ..., bypass-attention subnetwork D. In the cross-attention sub-network 1, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function, and the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, value vector and query vector are cross-attention calculated to obtain a first attention processing result 1; in the bypass attention sub-network 1, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function, and the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, value vector and query vector are cross-attention calculated to obtain a second attention processing result 1. The first attention processing result 1 and the second attention processing result 1 are weighted by the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result 1. Furthermore, the computer device can determine the cross-attention weighted result 1 as the query vector in the cross-attention sub-network 2 and the bypass attention sub-network 2, respectively, and use the fused coding feature as the key vector and value vector of the cross-attention function. Based on the cross-attention sub-network 2, a first attention processing result 2 is obtained, and based on the bypass attention sub-network 2, a second attention processing result 2 is obtained, and then a cross-attention weighted result 2 is obtained. And so on, until the processing of D cross-attention sub-networks and D bypass attention sub-networks is completed, a cross-attention weighted result D is obtained, and the target prediction noise is determined based on the cross-attention weighted result D. It should be noted that the key vector and value vector in the D cross-attention sub-networks and the D bypass attention sub-networks are both fused coding features.

[0152] The computer device can determine a target latent vector based on the target prediction noise and the target forward noise vector. Specifically, the computer device denoises the target forward noise vector using the target prediction noise to obtain the target latent vector. The computer device can then perform image reconstruction on the target latent vector using a decoder in the target video generation model to obtain a target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image in the to-be-generated video, located within the service image. Furthermore, the computer device obtains the next consecutive frame text corresponding to the target frame text from P consecutive frame texts, determines the next consecutive frame text as the new target frame text, and generates a new target predicted image corresponding to the new target frame text based on the target video generation model. When all P consecutive frame texts pass through the target predicted image corresponding to the target video generation model, i.e., P frame images are obtained, the computer device can determine the to-be-generated video containing the P frame images as the service video. The target latent vector is used to characterize the characteristics of the target predicted image, i.e., the latent representation of the target predicted image in the latent space after noise removal.

[0153] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business image and business text play a role in the process of generating the target prediction image, which makes the generated target prediction image and the historical image (i.e., the newly generated frame and historical frame in the video to be generated) have the same content (subject object and object scene, etc.), avoiding the excessive difference in the generated image due to the change of the text of the previous and next frames, improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text and composed of multiple target prediction images; at the same time, the constraints of historical text and historical image on the generation of target prediction image by the target video generation model can make the generation style or content of the target prediction image biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby further improving the overall tone consistency of the video and ensuring the quality of the video.

[0154] Further, see Figure 4 , Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will be described by taking the data processing method executed by a computer device as an example. Among them, the data processing method may at least include the following steps S401-S407:

[0155] Step S401: Obtain target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer.

[0156] In the embodiment of the present application, the specific implementation process of step S401 can be found in Figure 3 The specific description process of step S301 in will not be repeated here.

[0157] Step S402: obtain a noise data pool, determine P noise images from the noise data pool, obtain a target frame text traversed in sequence from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images.

[0158] In an embodiment of the present application, a computer device can obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer. The noise data can be strip noise data. The computer device randomly obtains M noise data from the noise data pool, splices the M noise data to obtain a first spliced ​​noise map, performs Fourier transform on the first spliced ​​noise map, obtains a first frequency domain corresponding to the first spliced ​​noise map, and determines the frequency signal in the first frequency domain that is less than the low-frequency threshold as a global low-frequency signal; M is a positive integer less than S. The specific value of the low-frequency threshold can be determined by the video generation task corresponding to the target text. For example, if the generation task corresponding to the target text is a scene-type video generation task, the low-frequency threshold can be determined to be 5 Hz, and the global low-frequency signal is a frequency signal with a frequency less than 5 Hz; the low-frequency threshold of the person or pet video generation task can be determined to be 10 Hz.

[0159] For details, please refer to Figure 5 , Figure 5 This is a noise image acquisition scenario provided by the embodiment of the present application. Figure 1 ,like Figure 5As shown, the computer device can randomly obtain M noise data from the noise data pool 501. For example, if M is 3, the computer device randomly obtains 3 noise data from the noise data pool 501 (for example, when S is greater than or equal to 7, the 3 randomly obtained noise data can be: noise data 1, noise data 5, and noise data 7). The 3 noise data are spliced ​​to obtain a first spliced ​​noise map, and the first spliced ​​noise map is Fourier transformed to obtain a first frequency domain corresponding to the first spliced ​​noise map. The frequency signal in the first frequency domain that is less than the low-frequency threshold is determined as a global low-frequency signal (that is, a frequency signal that is less than the low-frequency threshold is obtained from the first frequency domain and determined as a global low-frequency signal). The Fourier transform can be a 3D Fourier transform (3D FT). A 3D Fourier transform can represent a three-dimensional function or signal as the sum of its component frequencies. For images, it means converting information in the spatial domain to the frequency domain. In the frequency domain, low-frequency signals are usually concentrated in the low-frequency part of the transform result, that is, the area close to the origin. The computer device can extract these low-frequency components (ie, global low-frequency signals) by setting a low-frequency threshold or using a low-pass filter.

[0160] Furthermore, the computer device may obtain a first sliding step length, obtain P noise data groups from the S noise data using the first sliding step length, and splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M. Figure 5As shown, when M is equal to 3, the first sliding step is equal to 1, and the first noise data group in the P noise data groups is composed of noise data 1, noise data 2, and noise data 3 in the noise data pool, the second noise data group in the P noise data groups can be composed of noise data 2, noise data 3, and noise data 4 in the noise data pool. The computer device can perform Fourier transform on the P second spliced ​​noise images respectively to obtain the second frequency domains corresponding to the P second spliced ​​noise images, and determine the frequency signals greater than or equal to the low-frequency threshold in the P second frequency domains as high-frequency signals (that is, obtain frequency signals greater than or equal to the low-frequency threshold from the P second frequency domains, and determine them as P high-frequency information, including high-frequency signal 1, high-frequency signal 2, high-frequency signal 3, ..., high-frequency signal P). For example, high-frequency signal 1 can be the frequency signal of the high-frequency part (the part greater than or equal to the low-frequency threshold) obtained by the computer device from the first second frequency domain of the P second frequency domains. The computer device can combine the high-frequency signal in each second frequency domain with the global low-frequency signal to obtain P full-frequency signals, and perform inverse Fourier transform on each of the P full-frequency signals to obtain P noise images (including noise image 1a, noise image 2a, noise image 3a, ..., noise image Pa). For example, the computer device can combine the high-frequency signal 1 with the global low-frequency signal to obtain the full-frequency signal 1, and then perform inverse Fourier transform on the full-frequency signal 1 to obtain the noise image 1a. The inverse Fourier transform can be a 3D inverse Fourier transform (3D IFFT), which is used to convert a signal or function from the frequency domain back to its original time domain or space domain. In the one-dimensional case, the inverse Fourier transform is the process of converting frequency domain data back to the time domain; in the multi-dimensional case, such as the 3D IFFT, the inverse Fourier transform converts the three-dimensional frequency domain data back to the three-dimensional space domain.

[0161] The computer device can obtain the target frame text that is traversed in order from P consecutive frame texts, and determine the target noise image associated with the target frame text from the P noise images. For example, the P noise images can be sequentially associated with the consecutive frame texts in the P consecutive frame texts, where the consecutive frame text C1 (the first consecutive frame text) is associated with the noise image 1a, the consecutive frame text C2 is associated with the noise image 2a, ..., the consecutive frame text C P(The last continuous frame text) is associated with the noise image Pa. Then, when the target frame text is the first continuous frame text among P continuous frame texts, the target noise image can be the noise image 1a; when the target frame text is the second continuous frame text among P continuous frame texts, the target noise image can be the noise image 2a. The noise image obtained in this way has a consistent low-frequency signal, thereby ensuring that the subsequently generated target prediction image can maintain the layout environment and style consistency. The inconsistency of the high-frequency signal of the noise image allows the target video generation model to generate each frame image (corresponding to the target prediction image) when generating different editing image content according to the control information (target frame text, business image and business text), thereby achieving the strengthening of the initialization style layout control and improving the overall tone consistency of the video by maintaining the low frequency and randomly initializing the high frequency.

[0162] Optionally, the computer device may obtain the P noise images by obtaining a noise data pool and a second sliding step size, obtaining P noise data groups from the S noise data using the second sliding step size, and concatenating the noise data in the P noise data groups to obtain the P noise images. The noise data pool includes S noise data, and S is a positive integer.

[0163] Step S403: input the target noise image into the target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.

[0164] In the embodiment of the present application, the specific implementation process of step S403 can be found in Figure 3 The detailed description process of step S303 in the above is not repeated here. Figure 6 , Figure 6 This is a schematic diagram of a target video generation model structure provided by an embodiment of the present application. Figure 6 As shown, the computer device inputs the target noise image into the target video generation model, and extracts the target noise image features through the target video generation model to obtain the target noise feature Z. The target noise feature Z is further forward diffused for T time steps (time step length) through the forward diffusion network layer FD to obtain the target forward noise vector Z output by the forward diffusion network layer FD. T Among them, forward diffusion can refer to the process of adding noise to the target noise feature Z, and backward diffusion can refer to the process of adding noise to the target forward noise vector Z. T The time step is used to indicate the amplitude of the added noise in the forward diffusion and the amplitude of the predicted noise in the backward diffusion.

[0165] Step S404: input the business text, business image and target frame text into the target video generation model, and perform text encoding on the target frame text through the target video generation model to obtain text encoding features; the business image belongs to the frame image that has been generated in the video to be generated; the business text is the continuous frame text used to represent the business image among P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0166] In the embodiment of the present application, the specific implementation process of step S404 can be found in Figure 3 The specific description process of obtaining the text encoding feature in step S304 is not repeated here. Figure 6 ,like Figure 6 As shown, a computer device inputs the business text, business image, and target frame text into the target video generation model. The target frame text is encoded using the text encoding layer τ to obtain text encoding features. The business image is a frame image that has already been generated in the video to be generated, and the business image can be the last frame image in the video to be generated. The business text is the continuous frame text among P continuous frame texts that represents the business image, and the business text is the continuous frame text preceding the target frame text.

[0167] Step S405: perform feature extraction on the business image through the image encoding component in the target video generation model to obtain image extraction features; perform feature extraction on the business text through the text encoding component in the target video generation model to obtain text extraction features; map the image extraction features and the text extraction features to the same feature space, align the mapped image extraction features and the mapped text extraction features to obtain image-text encoding features.

[0168] In an embodiment of the present application, the target video model includes a multimodal feature extraction network, which includes an image encoding component, a text encoding component, and a feature adaptation component. The computer device can extract features from the business image through the image encoding component in the target video generation model to obtain image extraction features. Specifically, the image encoding component can be the Vision Transformer (ViT) in the BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) model. The BLIP model is a bootstrapped multimodal pre-training model for unified vision-language understanding and generation. The computer device divides the business image into B image blocks through ViT, linearly projects the B image blocks respectively, obtains B image projection features, adds corresponding image position features to the B image blocks in the B image projection features, obtains B image update features, and performs feature fusion on the B image update features to obtain image extraction features corresponding to the business image. Among them, the image position feature is used to characterize the position information of the image block in the business image, that is, the spatial feature. The image position feature can be a fixed-length vector, in which each element identifies whether the image block is located at a specific position. For example, if the business image is divided into a 2x2 block grid, the length of the position encoding vector corresponding to the image position feature will be 4, and the four elements indicate whether the image block is located in the corresponding rows and columns in the image. The image position feature of the image block in the upper left corner of the business image is [1, 0, 1, 0], indicating that it is in the first row and first column of the business image; the image position feature of the image block in the upper right corner is [1, 0, 0, 1], indicating that it is in the first row and second column of the business image; the image position feature of the image block in the lower left corner is [0, 1, 1, 0], indicating that it is in the second row and first column of the business image; the image position feature of the image block in the lower right corner is [0, 1, 0, 1], indicating that it is in the second row and second column of the business image.

[0169] The text encoding component can be an encoder-decoder (Bidirectional Encoder Representations from Transformers, BERT) based on the self-attention model in the BLIP model. The computer device can use BERT to split the business text to obtain a word-gram sequence corresponding to the business text (including G words, G is a positive integer), and perform vector conversion on each word-gram in the word-gram sequence corresponding to the business text to obtain word embedding vectors corresponding to the G words. Text position information is added to each word embedding vector to obtain G word update vectors. Among them, text position information refers to the information about the position of a word-gram in the business text. Feature fusion is performed on the G word update vectors to obtain text extraction features. Among them, the business image belongs to the frame image that has been generated in the video to be generated; the business text is the continuous frame text used to represent the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0170] The computer device can map the image extraction features and the text extraction features to the same feature space through a feature adaptation component, and perform feature alignment on the mapped image extraction features and the mapped text extraction features to obtain image-text coding features. The image-text coding features and the text coding features corresponding to the target frame text have the same feature dimensions. The feature alignment method can be to ensure that the associated image extraction features and text extraction features are close to each other in the feature space by normalization, scaling, or using a specific loss function (for example, triple loss or contrast loss).

[0171] For details, please refer to Figure 6 ,like Figure 6 As shown, the target video generation model includes a multimodal feature extraction network, which includes a multimodal feature extraction component MM (including an image encoding component and a text encoding component) and a feature adaptation component (such as Figure 6 The computer device can extract features from the business image through the image encoding component in the multimodal feature extraction component MM to obtain image extraction features; extract features from the business text through the text encoding component to obtain text extraction features; further map the image extraction features and text extraction features to the same feature space through the MLP, and align the mapped image extraction features with the mapped text extraction features to obtain image-text encoding features. The MLP can be composed of three stacked linear plus activation layers (single-layer perceptron). The specific layer composition can be seen in Table 1:

[0172] Table 1

[0173] Layer Name Layer Type Number of layers Linear1+act Linear layer 1 Dropout1 Activation layer (0.2) 1 Linear2+act Linear layer 1 Dropout2 Activation layer (0.2) 1 Linear3+act Linear layer 1 Dropout3 Activation layer (0.2) 1

[0174] As shown in Table 1, MLP includes 3 linear layers (Linear1+act, Linear2+act, Linear3+act) and 3 activation layers (Dropout1, Dropout2, Dropout3). The linear layer usually consists of a weight matrix and a bias vector to perform linear transformation. "Act" is used to indicate that an activation function is applied after the linear layer. The activation function can be ReLU, Sigmoid, or Tanh, etc., which is used to introduce nonlinearity so that the model can learn more complex function mappings. The Dropout layer is a regularization technique used to prevent neural networks from overfitting. In this layer, 20% of the neurons (i.e., a ratio of 0.2) are randomly dropped (i.e., set to zero) in each iteration.

[0175] Step S406, generating a fused coding feature based on the text coding feature and the graphic coding feature; in the back diffusion network layer in the target video generation model, based on the fused coding feature, the target forward noise vector is back diffused in T time steps respectively to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, a target latent vector is determined; the target latent vector is image reconstructed by the decoder in the target video generation model to obtain a target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image located in the business image in the video to be generated.

[0176] In an embodiment of the present application, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text. The specific implementation process of the computer device generating the fused coding feature based on the text coding feature and the image-text coding feature can be: the computer device performs feature splicing on the text coding feature and the image-text coding feature to obtain the spliced ​​coding feature, and performs dimension fusion on the spliced ​​coding feature through the feature fusion network in the target video generation model to obtain the fused coding feature. The fused coding feature has the same dimension as the target forward noise vector. For example, the feature fusion network can be a 1×1 convolutional network (Network), and the feature dimensions of the text coding feature and the image-text coding feature are both a, where a is a positive integer. After the computer device performs information splicing on the text coding feature and the image-text coding feature, it obtains a spliced ​​coding feature with a feature dimension of 2a. The computer device can perform cross-channel information interaction through the Network, reduce the dimension of the spliced ​​coding feature, and obtain a fused coding feature with a feature dimension of a. The fused coding feature has the same dimension as the target forward noise vector. As Figure 6As shown, the computer device can input the text encoding features output by the text encoding layer τ and the image-text encoding features output by the feature adaptation component MLP into the feature fusion network Network, perform dimension fusion on the spliced ​​encoding features through the Network, and output the fused encoding features.

[0177] T time steps include time step T i , i is a positive integer less than or equal to T; further, the computer device can be in the time step T of the back diffusion network layer in the target video generation model i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector.

[0178] Specifically, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each of which includes a cross attention sub-network and a bypass attention sub-network. When T is 3, the T noise prediction networks may include noise prediction network 1, noise prediction network 2, and noise prediction network 3. Noise prediction network 1, noise prediction network 2, and noise prediction network 3 each include a cross attention sub-network and a bypass attention sub-network. The computer device performs the following operations at time step T: i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The specific implementation process of the corresponding initial noise vector can be: the computer device at time step T i In the fusion encoding feature and iterative noise vector, input to the time step T i The corresponding noise prediction network; through time step T iThe cross-attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain a first attention processing result. Specifically, the computer device can use the initial noise vector as the query vector in the cross-attention function, the fused coding features as the key vector in the cross-attention function, and the fused coding features as the value vector in the cross-attention function. The computer device determines the product of the query vector and the key vector as the first fused feature through the cross-attention function, obtains the number of dimensions corresponding to the fused coding feature, determines the product of the first fused feature and the inverse of the number of dimensions as the second fused feature, converts the second fused feature into the first activation feature based on the activation sub-function in the cross-attention function, and determines the product of the first activation feature and the value vector as the first attention processing result. Among them, a possible cross-attention function can be seen in formula ①:

[0179]

[0180] As shown in formula ①, Attention(Q,K,V) represents the cross attention function, Q is used to represent the query vector query, K is used to represent the key vector key, and V is used to represent the value vector value; T represents transposition, K T represents the transposed matrix of K, d k Used to indicate the number of dimensions corresponding to the fused encoding features; is the first activated feature, The result of the first attention processing.

[0181] Furthermore, the computer device passes the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused encoded features and the iterative noise vector to obtain the second attention processing result. The specific implementation process of obtaining the second attention processing result can be referred to the process of obtaining the first attention processing result, and will not be repeated here.

[0182] The computer device can obtain the first cross-attention coefficient and the second cross-attention coefficient, and perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result. Further, based on the cross-attention weighted result, determine the time step T i The corresponding initial noise vector. A possible weighted processing process can be seen in formula ②:

[0183] W=αx1+βx2②

[0184] As shown in formula ②, W is used to represent the cross-attention weighted result, α is used to represent the first cross-attention coefficient, x1 is used to represent the first attention processing result, β is used to represent the second cross-attention coefficient, and x2 is used to represent the second attention processing result.

[0185] The computer device can convert the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise. i+1 is the time step T among T time steps i The next time step.

[0186] It should be noted that the number of cross attention sub-networks and bypass attention sub-networks in the noise prediction network corresponding to each time step can be multiple. Please refer to Figure 6 ,like Figure 6 As shown, the target video generation model includes a back diffusion network layer BD, and the computer device generates the target forward noise vector Z through the back diffusion network layer BD. T Perform back diffusion processing (denoising processing) for T time steps to obtain the target prediction noise Z0 output by the back diffusion network layer BD. In the first time step (i.e., time step T1), the noise prediction network 1 includes a first attention network ∈1 and a second attention network ω1. Among them, the first attention network ∈1 can include multiple cross attention sub-networks, and the second attention network ω1 can include multiple bypass attention sub-networks. For example, the first attention network ∈1 can include D (D can be equal to 16) cross attention sub-networks (Note: Figure 6 Only four cross-attention sub-networks are drawn, namely cross-attention sub-network U1, cross-attention sub-network U2, cross-attention sub-network U3, and cross-attention sub-network U4). The second attention network ω1 can include D (D can be equal to 16) bypass attention sub-networks ( Figure 7Only four bypass attention sub-networks are drawn in the figure, namely bypass attention sub-network V1, bypass attention sub-network V2, bypass attention sub-network V3, and bypass attention sub-network V4. Taking the four cross-attention sub-networks and the four bypass attention sub-networks as an example, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function in the cross-attention sub-network U1, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, value vector, and query vector are cross-attention calculated to obtain a first attention processing result 1c; the computer device can use the fused coding features as the key vector and value vector of the cross-attention function in the bypass attention sub-network V1, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, value vector, and query vector are cross-attention calculated to obtain a second attention processing result 1d. The first attention processing result 1c and the second attention processing result 1d are weighted by the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result 1e. Furthermore, the computer device can determine the cross-attention weighted result 1e as the query vector in the cross-attention sub-network U2 and the bypass attention sub-network V2 respectively, and use the fused coding features as the key vector and value vector of the cross-attention function, obtain the first attention processing result 2c based on the cross-attention sub-network U2, and obtain the second attention processing result 2d based on the bypass attention sub-network V2, and then obtain the cross-attention weighted result 2e, and so on, until the processing of D (for example, 4) cross-attention sub-networks and D bypass attention sub-networks is completed, and the cross-attention weighted result De is obtained. It should be noted that the key vectors and value vectors in the D cross-attention sub-networks and the D bypass attention sub-networks are all fused coding features. The computer device determines the initial noise vector Z corresponding to the first time step based on the cross-attention weighted result De. T-1 . Further, in the remaining T-1 time steps, continue through the first attention network ∈ i and the second attention network ω i The initial noise vector Z corresponding to the first time step T-1 Backward diffusion processing is performed until the target prediction noise Z0 is obtained. It is understandable that the computer device can reuse the existing SD model and change the input and add the bypass attention sub-network on the basis of the SD model to obtain the target video generation model. Therefore, when training the model, only the model parameters in the bypass attention sub-network need to be adjusted to obtain the target video generation model, without adjusting the model parameters in the cross attention sub-network in the SD model, thereby saving resources.

[0187] Among them, the noise prediction network includes several downsampling network layers, one intermediate sampling network layer and several upsampling network layers, such as Figure 6 The cross attention sub-network U1, cross attention sub-network U2, bypass attention sub-network V1, bypass attention sub-network V2 shown in the figure belong to the downsampling network layer, and the cross attention sub-network U3, cross attention sub-network U4, bypass attention sub-network V3, bypass attention sub-network V4 shown in the figure belong to the upsampling network layer. Please also refer to Figure 7 , Figure 7 This is a schematic diagram of a downsampling network layer structure provided by an embodiment of the present application, such as Figure 7 As shown, the downsampling network layer includes 2 residual processing layers (ResnetBlock2D), 2 transformation processing layers (Transformer2DModel) and 1 downsampling processing unit (DownSample2D). The computer device forwards the target noise vector Z T, fused coding features and time embedding vectors (used to characterize the number of time steps, here taking the first time step as an example), input into the first residual processing layer in the downsampling network layer, obtain the output result of the first residual processing layer, further input the output result into the first transform processing layer, obtain the output result of the first transform processing layer, fused coding features and time embedding vector, input into the second residual processing layer, obtain the output result and fused coding features of the second residual processing layer, and input into the second transform processing layer, input the input result of the second transform processing layer into the downsampling processing unit. The output result of the downsampling processing unit (that is, the output result of the downsampling processing network) is used to input into the next downsampling processing network for subsequent data processing. Among them, the main function of the residual processing layer is to help the downsampling network layer solve the gradient disappearance problem, especially in deep networks. By adding residual connections, the downsampling network layer can more easily learn the identity mapping, that is, directly pass the input to the output without losing information in the deep network. A typical ResnetBlock2D contains two convolutional layers, each of which is usually followed by batch normalization and ReLU activation functions. After the input of ResnetBlock2D passes through these layers, its output is added to the input to form a residual connection. In image segmentation tasks, Transformer2DModel allows the target video generation model to consider global context information when processing images through the self-attention mechanism. In the noise prediction network (U-net), Transformer2DModel can be used as part of the encoder or decoder, or as a bridge between the two to provide richer feature representations and context information. DownSample2D is used to reduce the spatial dimension of the input data, that is, the height and width of the image, while possibly increasing the number of channels.

[0188] Please also refer to Figure 8 , Figure 8 This is a schematic diagram of a transformation processing layer structure provided by an embodiment of the present application. Figure 8 As shown, the change processing layer includes 1 normalization processing layer (GroupNorm), 2 convolution processing layers (Conv1×1), 2 exchange processing layers and 1 transformation processing unit (BasicTransformerBlock). Among them, the input of the transformation processing layer can be as follows Figure 7The output of the residual processing layer is shown. The input of the transform processing unit is the fused encoded features and the output of the first exchange processing layer. The normalization processing layer (GroupNorm) is used to divide the channels corresponding to each input feature map (noise image) into multiple groups, and then normalize them within each group. Conv1×1 convolution is usually used to reduce or increase the number of channels of the feature map without changing the spatial dimensions (height and width) of the feature map. The exchange processing layer includes the permute function and the reshape operation, where the permute function is used to rearrange the dimensions of the tensor (feature or vector). For example, a tensor of shape (N, C, H, W), where N is the batch size, C is the number of channels, H is the height, and W is the width, can be rearranged to (N, H, W, C) using permute. This rearrangement can convert the image data from a two-dimensional spatial feature map into a serialized form, which is useful for inputting into the transform processing unit. The reshape operation is used to change the shape of the tensor without changing its data. In the transformation processing layer, the reshape operation is often used to convert the serialized feature map back to its original two-dimensional or three-dimensional shape for further processing or combination with other parts of the U-Net.

[0189] The structure of the transformation processing unit can be found in Figure 9 , Figure 9 This is a schematic diagram of a transformation processing unit structure provided by an embodiment of the present application. Figure 9 As shown in Figure 2, the transform processing unit includes three linear processing layers (LayerNorm), a self-attention processing layer (SelfAttention), a cross-attention processing layer (CrossAttention), a bypass attention processing layer and a feedforward processing layer. The input of the transform processing unit can be as follows: Figure 8 The output of the first exchange processing layer shown. Among them, the linear processing layer is used to perform a linear transformation on the input to obtain a linear output. The self-attention processing layer is used to capture the relationship between any two elements in the input sequence. The cross-attention processing layer and the bypass attention processing layer are both used to capture the attention relationship between two sequences (the fused coding features and the output of the second linear processing layer), that is, the attention relationship between the target forward noise vector or the iterative noise vector and the fused coding features. The computer device can perform weighted processing on the cross-attention processing result obtained by the cross-attention processing layer and the bypass attention processing result obtained by the bypass attention processing layer to obtain an attention weighted result, and input it into the third linear processing layer of the transformation processing unit, and input the output result of the linear processing layer into the feedforward processing layer. The feedforward processing layer can be used to perform feature transformation on the input sequence, introduce a nonlinear activation function (for example, a ReLU function), and perform nonlinear processing.

[0190] Furthermore, the computer device predicts the noise Z0 and the target forward noise vector Z T Determine the target latent vector, reconstruct the target latent vector through the decoder D in the target video generation model, and obtain the target prediction image Y corresponding to the target frame text. Use the target prediction image Y as the next frame image located in the business image in the video to be generated. Furthermore, the computer device obtains the next continuous frame text corresponding to the target frame text in the P continuous frame texts, determines the next continuous frame text as the new target frame text, and generates a new target prediction image corresponding to the new target frame text based on the target video generation model. When the P continuous frame texts all pass through the target prediction image corresponding to the target video generation model, that is, P frame images are obtained, the computer device can obtain the video to be generated including P frame images, wherein the P frame images include the target prediction image Y.

[0191] Step S407: performing frame interpolation processing between every two adjacent frame images in the video to be generated.

[0192] In an embodiment of the present application, after obtaining the video to be generated, the computer device can also perform interpolation processing between each two adjacent frame images of the video to be generated to further increase the continuity of the video. Specifically, when the video to be generated contains frame images corresponding to P consecutive frame texts, the computer device performs image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images. Among them, the frame images corresponding to the P consecutive frame texts include the target prediction image. Among them, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P. The computer device can be based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result. j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image.

[0193] Specifically, the computer device can use the optical flow algorithm to calculate the frame image Pj-1 To frame image P j The optical flow vector field and frame image P j+1 To frame image P j+2 The optical flow vector field is used to describe the movement of pixel points in space, such as the frame image P j-1 To frame image P j The optical flow vector field is used to describe the frame image P j-1 The pixel points in the frame image P j-1 The corresponding time to frame image P j The corresponding moment, the operation in space. The computer device can use the quadratic optical flow model based on the frame image P j-1 To frame image P j The optical flow vector field and frame image P j+1 To frame image P j+2 The optical flow vector field is calculated by calculating the pixel points from the frame image P j The corresponding time to frame image P j+1 The optical flow vector field of a certain moment in the corresponding moment is used to predict the position of the pixel point corresponding to that moment (i.e. the pixel point position of the intermediate frame). Further, the computer device uses a differentiable optical flow reversal layer to predict the reverse optical flow, that is, from the intermediate frame back to the frame image P j and from the intermediate frame to the frame image P j+1 The optical flow vector field (i.e., the optical flow prediction result) of the frame image P j , frame image P j+1 And the optical flow prediction results are used to generate the frame image P through image reprojection and interpolation technology. j and frame image P j+1 The computer device may also apply an adaptive flow filter to the reversed optical flow (reverse optical flow) to reduce possible ringing effects and improve the quality of the intermediate frames. Furthermore, the computer device determines the video to be generated, which includes the frame images corresponding to the P consecutive frames of text and the intermediate frame images, as a business video that matches the target text.

[0194] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business image and business text play a role in the process of generating the target prediction image, which enables the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) to maintain consistency in content (subject object and object scene, etc.), avoiding excessive differences in the generated images due to changes in the previous and next frame texts, and improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text and composed of multiple target prediction images. At the same time, the high-frequency signal of the frame-by-frame noise map (i.e., the first spliced ​​noise map) generated by shifting the noise data pool is combined with the global low-frequency signal of the noise data pool as the frame-by-frame noise image for initialization. The consistency of the low-frequency signal can maintain the layout environment and style consistency. The inconsistency of the high-frequency signal allows the target video generation model to generate different edited image content for each frame according to the control information (target frame text, business image, and business text). This further strengthens the initialization style layout control and improves the overall tone consistency of the video by maintaining the low frequency and performing high-frequency random initialization. In addition, an intermediate frame image is inserted between every two adjacent frame images in the video to be generated to further improve the coherence of the business video indicated by the target text and the consistency of the overall tone of the video, thereby ensuring the quality of the generated video.

[0195] Further, see Figure 10 , Figure 10 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will be described by taking the data processing method executed by a computer device as an example. Among them, the data processing method may at least include the following steps S1001-S1007:

[0196] Step S1001: Acquire target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer.

[0197] In the embodiment of the present application, the specific implementation process of step S1001 can be found in Figure 4 The specific description of step S401 is omitted here.

[0198] Step S1002: obtain a noise data pool, obtain P noise data groups from the S noise data in the noise data pool through a second sliding step, splice the noise data in the P noise data groups respectively to obtain P noise images, obtain the target frame text traversed in sequence from the P consecutive frame texts, and determine the target noise image associated with the target frame text from the P noise images.

[0199] In an embodiment of the present application, a computer device may obtain a noise data pool, wherein the noise data pool includes S noise data, where S is a positive integer. The computer device may obtain a second sliding step size, and using the second sliding step size, obtain P noise data groups from the S noise data, and concatenate the noise data in the P noise data groups to obtain P noise images. Each noise data group includes M noise data, where M is a positive integer, and the second sliding step size is less than M.

[0200] Specifically, the specific implementation method of the computer device obtaining P noise images can be found in Figure 11 , Figure 11 This is a noise image acquisition scenario provided by the embodiment of the present application. Figure 2 ,like Figure 11 As shown, the computer device can obtain the noise data pool 1101 (with Figure 5 The noise data pool 1101 includes S noise data; the computer device can obtain a second sliding step length, and obtain P noise data groups from the S noise data through the second sliding step length. The specific implementation process of obtaining the P noise data groups can be the same as Figure 5 The number of noise data included in each noise data group is M, for example Figure 6 Where M = 3, the computer device can respectively splice the noise data in the P noise data groups to obtain P noise images, including noise image 1b, noise image 2b, noise image 3b, ..., noise image Pb. The second sliding step is less than M, which means that at least one noise data is the same between two adjacent noise data groups in the P noise data groups. For example, if the second sliding step is 2 and M = 3, the first noise data group in the P noise data groups can be composed of noise data 1, noise data 2, and noise data 3; the second noise data group in the P noise data groups can be composed of noise data 3, noise data 4, and noise data 5; and the first noise data group and the second noise data group both contain the same noise data 3.

[0201] The computer device obtains the target frame text that is traversed in sequence from the P consecutive frame texts, and determines the target noise image associated with the target frame text from the P noise images. For example, the P noise images can be sequentially associated with the consecutive frame texts in the P consecutive frame texts, where the consecutive frame text C1 (the first consecutive frame text) is associated with the noise image 1b, the consecutive frame text C2 is associated with the noise image 2b, ..., the consecutive frame text C P (The last continuous frame text) is associated with noise image Pb. When the target frame text is the first continuous frame text among P continuous frame texts, the target noise image can be noise image 1b; when the target frame text is the second continuous frame text among P continuous frame texts, the target noise image can be noise image 2b. In this way, two adjacent noise images among the P noise images obtained have some of the same noise data, and the remaining noise data varies randomly, thereby generating an initialization noise map (noise image) that maintains a certain proportion of the same initialization. This avoids excessive differences in the final style of the generated frames due to initialization differences between consecutive frames, thereby improving the overall consistency of the video tone.

[0202] Optionally, the specific implementation process of the computer device obtaining P noise images can also be found in Figure 4 The specific description of determining P noise images from the noise data pool in step S402 shown in FIG.

[0203] Step S1003: input the target noise image into the target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.

[0204] In the embodiment of the present application, the specific implementation process of step S1003 can be found in Figure 4 The specific description of step S403 is omitted here.

[0205] Step S1004: input the business text, business image and target frame text into the target video generation model, and perform text encoding on the target frame text through the target video generation model to obtain text encoding features.

[0206] In the embodiment of the present application, the specific implementation process of step S1004 can be found in Figure 4 The specific description of step S404 is omitted here.

[0207] Step S1005, feature extraction is performed on the business image through the image encoding component in the target video generation model to obtain image extraction features; feature extraction is performed on the business text through the text encoding component in the target video generation model to obtain text extraction features; the image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are feature aligned to obtain image-text encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text in P continuous frame texts used to represent the business image, and the business text is the continuous frame text sorted before the target frame text.

[0208] In the embodiment of the present application, the specific implementation process of step S1005 can be found in Figure 4 The specific description content of step S405 is not repeated here.

[0209] Step S1006, the unit features in the F graphic coding sub-features are combined into a sequence to be matched; based on the sequence to be matched and the text coding features, the fused coding features are determined; in the back diffusion network layer in the target video generation model, based on the fused coding features, the target forward noise vector is back diffused in T time steps to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, the target latent vector is determined; the target latent vector is image reconstructed by the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image located in the business image in the video to be generated.

[0210] In an embodiment of the present application, the number of business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the video to be generated; the graphic coding feature includes F graphic coding sub-features, and one graphic coding sub-feature is obtained by multimodal feature extraction based on a business text and the represented business image.

[0211] For details, please refer to Figure 12 , Figure 12 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 2 ,like Figure 12As shown, the business text includes business text 1 to business text F, and the business image includes business image 1 to business image F. The computer device can extract features from the F business images using the image encoding component in the multimodal feature extraction component MM to obtain F image extraction features; and can extract features from the F business texts using the text encoding component to obtain F text extraction features. The computer device can further map the one-to-one corresponding image extraction features and text extraction features to the same feature space using the feature adaptation network (MLP), and align the mapped image extraction features with the mapped text extraction features to obtain F image-text encoding sub-features. The computer device can combine the unit features in the F image-text encoding sub-features into a sequence to be matched. Specifically, the computer device can bias the F image-text encoding sub-features using a bitwise addition structure (Add) to obtain F updated image-text sub-features with bias characteristics. The bias feature is used to represent the different value ranges of the biased feature, allowing the target video generation model to distinguish whether the feature is generated based on text or based on both image and text. The computer device obtains the unit features of each updated image and text sub-feature from the F updated image and text sub-features, and composes them into a sequence to be matched. Furthermore, the computer device can perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature. Among them, the computer device can obtain the feature matching degree between each unit feature and the text encoding feature by the Euclidean distance or cosine similarity method. The computer device can sort each unit feature based on the feature matching degree of each unit feature to obtain a feature matching sequence. The computer device can determine the first A unit features in the feature matching sequence as the target matching features; A is a positive integer. The computer device performs feature splicing on the text encoding features and the target matching features to obtain matching splicing features, and further performs dimension fusion on the matching splicing features through a feature fusion network (Network) to obtain a fused encoding feature. Among them, the Network is used to make the feature dimension of the fused encoding feature the same as the dimension of the target forward noise vector. For example, when the feature length of the text encoding feature is 77 and A=3, the 77th bit in the text encoding feature is not fully used and a portion of it is padded with default values. In this case, the computer device can determine the first three unit features in the feature matching sequence as the target matching feature and replace the target matching feature with the padded default values ​​of bits 74 to 76 in the text encoding feature to obtain a fused encoding feature. Bit 77 in the text encoding feature is the default end bit and is not used.

[0212] The computer device performs back diffusion processing on the target forward noise vector in the back diffusion network layer in the target video generation model based on the fusion coding features in T time steps to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, a target latent vector is determined; the target latent vector is reconstructed by the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image of the business image in the video to be generated. For the specific implementation process, please refer to Figure 4 The relevant description of step S406 shown in FIG will not be repeated here.

[0213] Step S1007: performing frame interpolation processing between every two adjacent frame images in the video to be generated.

[0214] In the embodiment of the present application, the specific implementation process of step S1007 can be found in Figure 4 The specific description of step S407 is omitted here.

[0215] It is understandable that Figure 12 For the specific implementation process of each network layer and various components in the model component 1201 in the target video generation model, please refer to the above Figure 4 The corresponding embodiments will not be described in detail here.

[0216] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model, generating a target predicted image for the target frame text. This allows the features of multiple business images and multiple business texts to play a role in the generation of the target predicted image. This ensures that the content (such as the subject and scene) of the generated target predicted image and the historical image (i.e., the previous and next frames of the generated video) remain consistent, avoiding large differences in the generated images due to changes in the previous and next frames' text. This improves the similarity between the target predicted image and the historical image, thereby enhancing the coherence and coordination of the generated video composed of multiple target predicted images based on the target text. Furthermore, frame-by-frame noise images generated by shifting the noise data pool are used as initialization, and adjacent noise images share a certain proportion of the same information. This prevents large differences in the final style of the generated frames due to initialization differences between consecutive frames generated based on the noise images, thereby strengthening the control of the initialization style layout and improving the overall tonal consistency of the video. Furthermore, an intermediate frame image is inserted between every two adjacent frames in the generated video, further improving the coherence and overall tonal consistency of the business video indicated by the target text, thereby ensuring the quality of the generated video.

[0217] See Figure 13 , Figure 13This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will be described by taking the data processing method executed by a computer device as an example. Among them, the data processing method may include at least the following steps S1301-S1306:

[0218] Step S1301 , obtaining a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer.

[0219] In an embodiment of the present application, a computer device can obtain a sample video, obtain N video frames based on the sorting order of the video frames in the sample video, and determine the N video frames as N image samples. Wherein, the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other. The computer device can generate text samples for the N image samples respectively through an image description generation model to obtain N text samples. Wherein, the text sample is used to indicate the specific content in the corresponding image sample. The image description generation model can be a generative adversarial network model (Generative Adversarial Networks, GAN), a BLIP model or a scholar multimodal large model (InternVL), or other models that can be used as image generation text descriptions, which are not limited here. The computer device can obtain N noise samples, combine the N image samples, N text samples and N noise samples to obtain a sample sequence. For example, a computer device obtains 10 video frames (image1 to image10) from a sample video, generates corresponding text samples (text1 to text10) for each of the 10 video frames, and obtains 10 noise samples (noise sample 1 to noise sample 10). The resulting sample sequence can be expressed as "(image1, text1, noise sample 1), (image2, text2, noise sample 2), (image3, text3, noise sample 3), ..., (image10, text10, noise sample 10)". The specific implementation process of the computer device obtaining N noise samples can be: the computer device obtains a noise data pool; the noise data pool includes S noise data; S is a positive integer. R noise data are randomly obtained from the noise data pool, the R noise data are concatenated to obtain a first sample noise map, the first sample noise map is Fourier transformed to obtain a first sample frequency domain corresponding to the first sample noise map, and a frequency signal in the first sample frequency domain that is less than a sample low-frequency threshold is determined as a sample global low-frequency signal; R is a positive integer less than S. Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data using the first sample sliding step length, concatenate the noise data in the N sample noise data groups, and obtain N second sample noise maps; each sample noise data group includes R number of noise data. Perform Fourier transform on each of the N second sample noise maps to obtain second sample frequency domains corresponding to the N second sample noise maps, and determine frequency signals greater than or equal to the sample low-frequency threshold in each of the N second sample frequency domains as sample high-frequency signals. Combine the sample high-frequency signals in each second sample frequency domain with the sample global low-frequency signal to obtain N sample full-frequency signals, and perform inverse Fourier transform on each of the N sample full-frequency signals to obtain N noise samples.The specific implementation process of the computer device obtaining N noise samples can also be found in FIG. Figure 4 In step S402 shown in FIG, P noise images are obtained, or as shown in FIG. Figure 5 The process of obtaining the noise image scene schematic diagram shown is not repeated here.

[0220] Step S1302 : determining a target image sample from the N image samples, and determining a target text sample associated with the target image sample from the N text samples.

[0221] In this embodiment of the present application, a computer device sequentially traverses N image samples to obtain a target image sample, and determines a target text sample associated with the target image sample from the N text samples. For example, the first target image sample obtained by the computer device from the N image samples is image1, and the target text sample associated with the target image sample determined from the N text samples is text1.

[0222] Step S1303 , performing feature extraction on the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain a latent noise feature, and performing forward diffusion processing on the latent noise feature to obtain a sample forward noise vector.

[0223] In the embodiment of the present application, the computer device can determine the target noise sample corresponding to the target image sample from N noise samples. For example, if the target image sample is image1, the target noise sample can be the first noise sample among the N noise samples. Figure 14 , Figure 14 This is a schematic diagram of the structure of an initial video generation model provided in the embodiment of the present application. Figure 1 ,like Figure 14 As shown, the computer device can input the target noise sample and the target image sample x into the initial video generation model, perform feature extraction on the target image sample through the encoder E (for example, it can be an image encoder VAE) in the initial video generation model to obtain the target image sample feature, perform feature extraction on the target noise sample through the initial video generation model to obtain the target noise sample vector, and determine the potential noise feature Z' based on the target image sample feature and the target noise sample vector. In the T time steps of the initial forward diffusion network layer FD, the computer device continuously adds the noise vector (target noise sample vector) corresponding to the target noise sample to the potential noise feature to obtain the sample forward noise vector Z T '.

[0224] Step S1304: perform text encoding on the target text sample to obtain a first encoding feature, and perform multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample.

[0225] In the embodiment of the present application, the computer device performs text encoding on the target text sample to obtain the specific implementation process of the first encoding feature, which can be seen in FIG. Figure 3 In step S304 shown in FIG, the target frame text is encoded to obtain the specific implementation process of the text encoding feature. The computer device extracts multimodal features from the first image sample and the first text sample to obtain the specific implementation process of the second encoding feature, which can be seen in FIG. Figure 3 In step S304 shown in , multimodal feature extraction is performed on business text and business image to obtain the specific implementation process of image-text coding features, which will not be repeated here. Among them, the first image sample is an image sample that is sorted before the target image sample in the N image samples, and is the previous image sample of the target image sample in the N image samples, and the first text sample is a text sample associated with the first image sample. It should be noted that when the target image sample is the first image sample in the N image samples, the first text sample and the first image sample input to the initial video generation model are null value inputs, that is, at this time the first text sample is a blank null value text, and the first image sample is a blank null value image. Please also refer to Figure 14 ,like Figure 14 As shown, the initial video generation model also includes a text encoding layer τ and an initial multimodal feature extraction network. The initial multimodal feature extraction network includes a multimodal feature extraction component MM (including an image encoding component and a text encoding component) and an initial feature adaptation component MLP. The computer device can perform text encoding on the target text sample through the text encoding layer τ to obtain a first encoding feature. The image encoding component in the multimodal feature extraction component MM performs feature extraction on the first image sample to obtain a first image extraction feature; the text encoding component performs feature extraction on the first text sample to obtain a first text extraction feature, and the first image extraction feature and the first text extraction feature are mapped to the same feature space. The mapped first image extraction feature and the mapped first text extraction feature are feature aligned to obtain an image-text extraction feature. Furthermore, the computer device performs feature adaptation on the image-text extraction feature through the initial feature adaptation component MLP to obtain a second encoding feature. The feature dimension of the second encoding feature is the same as that of the first encoding feature.

[0226] Step S1305 , performing reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector.

[0227] In the embodiment of the present application, the computer device can perform feature fusion on the first coding feature and the second coding feature to obtain the target coding feature. The specific implementation process can be seen in FIG. Figure 4 In step S406 shown in FIG, the specific implementation process of generating the fusion coding feature based on the text coding feature and the image-text coding feature is not described here. Figure 14 ,like Figure 14 As shown, the initial video generation model also includes an initial back diffusion network layer BD, which includes a first attention network ∈1 and a second initial attention network ω1. The first attention network ∈1 includes multiple cross attention sub-networks, and the second initial attention network ω1 includes multiple initial bypass attention sub-networks. For example, in the first time step (i.e., time step T1), the first attention network ∈1 may include D (D may be equal to 16) cross attention sub-networks (Note: Figure 14 Only four cross-attention sub-networks are drawn, namely cross-attention sub-network U1, cross-attention sub-network U2, cross-attention sub-network U3, and cross-attention sub-network U4). The second initial attention network ω1 can include D (D can be equal to 16) initial bypass attention sub-networks ( Figure 14 Only four initial bypass attention sub-networks are drawn, namely initial bypass attention sub-network O1, initial bypass attention sub-network O2, initial bypass attention sub-network O3, and initial bypass attention sub-network O4).

[0228] The computer device can use the target encoding feature as the key vector and value vector of the cross attention function in the cross attention sub-network U1, and the sample forward noise vector Z T 'As the query vector of the cross attention function, the cross attention calculation is performed on the key vector, the value vector and the query vector through the cross attention function to obtain the first sample attention result 1f; the computer device can use the target encoding feature as the key vector and value vector of the cross attention function in the initial bypass attention sub-network O1, and use the sample forward noise vector Z T'As the query vector of the cross-attention function, the cross-attention function is used to perform cross-attention calculation on the key vector, the value vector, and the query vector to obtain the second sample attention result 1q. The computer device can obtain the first cross-attention sample coefficient and the second cross-attention sample coefficient, and perform weighted processing on the first sample attention result 1f and the second sample attention result 1q through the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain the sample attention weighted result 1r. Further, the computer device can determine the sample attention weighted result 1r as the query vector in the cross-attention sub-network U2 and the initial bypass attention sub-network O2, respectively, and use the target encoding feature as the key vector and value vector of the cross-attention function to obtain the first sample attention result 2f based on the cross-attention sub-network U2 and the second sample attention result 2q based on the initial bypass attention sub-network O2, and then obtain the sample attention weighted result 2r through the first cross-attention sample coefficient and the second cross-attention sample coefficient, and so on, until the processing of D (for example, 16) cross-attention sub-networks and D initial bypass attention sub-networks is completed to obtain the sample attention weighted result Dr. It should be noted that the key vectors and value vectors in the D cross attention sub-networks and the D initial bypass attention sub-networks are all target encoding features. The computer device determines the initial sample noise vector Z corresponding to the first time step based on the sample attention weighted result Dr T-1 '. Further, in the remaining T-1 time steps, the computer device continues to use the first attention network ∈ i and the second initial attention network ω i For the initial sample noise vector Z corresponding to the first time step T-1 'Perform reverse diffusion processing until the predicted noise sample vector Z0 is obtained'.

[0229] Step S1306 , adjusting the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0230] In an embodiment of the present application, the initial video generation model also includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature; the initial feature fusion network is used to perform feature fusion on the first coding feature and the second coding feature to obtain a target coding feature. Among them, the initial multimodal feature extraction network includes a multimodal feature extraction component MM (including an image coding component and a text coding component) and an initial feature adaptation component MLP. The initial feature adaptation component MLP is used to perform feature adaptation on the image and text extraction features and the first coding features, so that the adapted image and text extraction features (i.e., the second coding features) have the same feature dimension as the first coding features.

[0231] After obtaining the predicted noise sample vectors corresponding to the N noise samples, the computer device can generate a model loss value based on the N noise samples and the N predicted noise sample vectors. The generation process of the model loss value can be referred to Formula ③:

[0232]

[0233] As shown in formula ③, L LDM It is used to represent the model loss value, E represents the expected value, ε(x) is used to represent the encoding result of the encoder E in the initial video generation model encoding the target image sample x, y is used to represent the initial video generation model, based on the target image sample x, the target sample prediction image y, ∈~Ν(0,1), is used to represent that the random noise ∈ obeys the standard normal distribution, that is, the Gaussian distribution with a mean of 0 and a variance of 1. In the embodiment of the present application, the random noise ∈ is the target noise sample. t is used to represent the time step, ∈ θ The predicted noise sample vector for the target noise sample output by the initial video generation model, z t Used to represent the sample forward noise vector (i.e. Figure 11 Z in T '), c is used to represent the target encoding feature. Represents the noise ∈ predicted by the initial video generation model θ The L2 loss between the (predicted noise sample vector) and the actual sampled noise ∈ (target noise sample). This loss measures the initial video generation model’s performance in the given sample forward noise vector z t , time step t, and target encoding feature c, the accuracy of noise prediction. L2 loss, also known as Euclidean loss or mean squared error (MSE), measures the sum of squares of the differences between the model's predicted values ​​and the actual observed values ​​(true values). This is the sum of squares of the differences between the target noise samples (actual observed values) and the predicted noise sample vectors (model predicted values) of the initial video generation model.

[0234] Furthermore, the computer device can adjust the parameters of the initial multimodal feature extraction network, the initial feature fusion network, and the initial bypass attention sub-network based on the model loss value to obtain a target video generation model. The target video model is used to generate a target video indicated by the target text based on the target text. Optionally, in the process of training the initial video generation model, the computer device can use multiple batches of noise samples, image samples, and text samples, set a certain learning rate (for example, 1e-4), and perform multiple rounds of training on the initial video generation model. When each round of training is completed, the model loss value corresponding to the round of training is used to adjust the parameters of the initial video generation model until multiple rounds of training are completed to obtain the final target video generation model.

[0235] Through the above process, a noise data pool is used to obtain N noise samples with consistent low-frequency signals and randomly generated high-frequency signals. The sample forward noise vector is further determined using the target noise sample. This allows the initial video generation model to learn how to obtain a predicted noise sample vector based on the sample forward noise vector and target coding features, so that it can generate an image similar to the target image sample. The target coding features are jointly determined by the target text sample, the first image sample (historical sample image), and the first text sample (historical sample text) as joint control information. This enables the initial video generation model to learn the association between the historical image and the current frame image (target image sample). This ensures that the trained target video generation model has the ability to generate consistent content between previous and next frames, and ensures that the features corresponding to the business image and business text are effective when the target video generation model predicts the target predicted image. This ensures that the target predicted image is consistent and coherent with the business image.

[0236] Moreover, the embodiment of the present application performs task adaptability training on the basis of the existing SD model, reuses the existing SD model, and transforms it on its basis to obtain an initial video generation model, that is, the input of the SD model is changed on the existing SD model, that is, from being controlled by the target text sample to being jointly controlled by the target text sample, the first text sample and the first image sample; and an initial bypass attention subnetwork is added to the existing SD model, that is, the existing SD model only includes the cross-attention subnetwork, and the initial video generation model includes the cross-attention subnetwork and the initial bypass attention subnetwork, which can enable the initial video generation model to have the generation capability of keeping the contents of the previous and next frames consistent, avoid the mismatch between the target video generation model and the target video generation task caused by non-training methods, and save training resources in the model training process.

[0237] See Figure 15 , Figure 15This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 5 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will be described by taking the data processing method executed by a computer device as an example. Among them, the data processing method may at least include the following steps S1501-S1506:

[0238] Step S1501: obtain a sample video, obtain N video frames based on the sorting order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same main object and object scene, and the display perspectives of the main objects in the N image samples are different; generate text samples for the N image samples respectively to obtain N text samples; the text samples are used to indicate the specific content in the corresponding image samples; obtain N noise samples, and combine the N image samples, N text samples and N noise samples to obtain a sample sequence.

[0239] In an embodiment of the present application, a computer device may obtain a sample video, obtain N video frames based on the sorting order of the video frames in the sample video, and determine the N video frames as N image samples. The N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different. The computer device may generate text samples for the N image samples respectively to obtain N text samples. The text samples are used to indicate the specific content in the corresponding image samples. The specific implementation process of the computer device obtaining N noise samples may be: the computer device obtains a noise data pool. The noise data pool includes S noise data; S is a positive integer. Further, the computer device may obtain a second sample sliding step, and obtain N noise data groups from the S noise data through the second sample sliding step, and splice the noise data in the N noise data groups to obtain N noise samples; wherein the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step is less than R. Optionally, the specific implementation process of the computer device obtaining N noise samples can also be found in. Figure 13 The computer device may combine N image samples, N text samples, and N noise samples to obtain a sample sequence.

[0240] Step S1502 : determining a target image sample from the N image samples, and determining a target text sample associated with the target image sample from the N text samples.

[0241] In the embodiment of the present application, the specific implementation process of step S1502 can be found in Figure 13 The specific description process of step S1302 shown in FIG will not be repeated here.

[0242] Step S1503 , performing feature extraction on the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain a latent noise feature, and performing forward diffusion processing on the latent noise feature to obtain a sample forward noise vector.

[0243] In the embodiment of the present application, the specific implementation process of step S1503 can be found in Figure 13 The specific description process of step S1303 shown in FIG will not be repeated here.

[0244] Step S1504: perform text encoding on the target text sample to obtain a first encoding feature, and perform multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample.

[0245] In the embodiments of this application, please refer to Figure 16 , Figure 16 This is a schematic diagram of the structure of an initial video generation model provided in the embodiment of the present application. Figure 2 ,like Figure 16As shown, the computer device can perform text encoding on the target text sample through the text encoding layer τ to obtain a first encoding feature. Wherein, the number of the first text samples and the first image samples is H, H is a positive integer, and the H first image samples are all image samples ranked before the target image sample in the N image samples, that is, the specific value of H is determined by the ranking order of the target image sample in the N image samples. For example, if the target image sample is the 5th image sample in the N image samples, then the value of H is 4, and the first image sample includes the first 4 image samples in the N image samples (first image sample 1, first image sample 2, first image sample 3, first image sample 4). The first text sample is the text sample associated with the first 4 image samples respectively, that is, the number of the first text samples is also 4, including the first text sample 1, the first text sample 2, the first text sample 3, and the first text sample 4. The second encoding feature includes H second encoding sub-features, and a second encoding sub-feature is obtained by performing multimodal feature extraction based on a first text sample and the represented first image sample. The computer device can perform multimodal feature extraction on a first text sample and a represented first image sample based on the multimodal extraction component MM in the multimodal feature extraction network in the initial video generation model to obtain a first image-text sample feature. The computer device can perform feature adaptation on the first image-text sample feature through the initial feature video component MLP in the multimodal feature extraction network to obtain an updated image-text sample feature. After obtaining H updated image-text sample features, the computer device can bias the H updated image-text sample features using a bitwise addition structure Add to obtain H second encoding sub-features with bias characteristics. The bias feature is used to represent the different value ranges of the biased feature, allowing the initial video generation model to distinguish whether the feature is generated based on text or based on both image and text.

[0246] Step S1505 : performing reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector.

[0247] In the embodiment of the present application, the computer device can generate the target coding feature according to the first coding feature and the second coding feature. The specific implementation process can be found in FIG. Figure 10 The specific description process of determining the fusion coding features in step S1006 shown in FIG will not be repeated here.

[0248] Furthermore, the computer device performs reverse diffusion processing on the sample forward noise vector in T time steps based on the target coding feature to obtain the specific implementation process of the predicted noise sample vector. Figure 13 The specific description process of step S1305 shown in FIG will not be repeated here.

[0249] Step S1506, adjusting the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0250] In the embodiment of the present application, the specific implementation process of step S1506 can be found in Figure 13 The specific description process of step S1306 shown in FIG will not be repeated here.

[0251] It is understandable that Figure 16 For the specific implementation process of each network layer and various components in the initial model component 1601 in the initial video generation model, please refer to the above Figure 13 The corresponding embodiments will not be described in detail here.

[0252] Through the above process, N noise samples are obtained from a noise data pool. Two adjacent noise samples in these N noise samples share a certain proportion of identical information. Furthermore, a sample forward noise vector is determined using a target noise sample from the N noise samples. This allows the initial video generation model to learn how to obtain a predicted noise sample vector based on the sample forward noise vector and target coding features, thereby enabling it to generate an image similar to the target image sample. The target coding features are jointly determined by the target text sample, the first image sample (historical sample image), and the first text sample (historical sample text) as joint control information. This allows the initial video generation model to learn the association between multiple historical images and the current frame image (target image sample). This enables the trained target video generation model to maintain consistent generation capabilities between the content of previous and subsequent frames, ensuring that the features corresponding to the business image and business text are effective when the target video generation model predicts the target predicted image. This ensures that the target predicted image is consistent and coherent with the business image.

[0253] Further, see Figure 17 , Figure 17 This is a schematic diagram of a data processing device provided in an embodiment of the present application. Figure 1 The data processing device 1700 may include: a data acquisition module 11 , a data determination module 12 , a data forward diffusion module 13 , a feature encoding module 14 , an image prediction module 15 , and a video determination module 16 .

[0254] The data acquisition module 11 is used to acquire the target text, split the target text, and obtain P consecutive frames of text in the order of video frames; P is a positive integer;

[0255] The data determination module 12 is configured to obtain P noise images, obtain a target frame text traversed in sequence from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images;

[0256] The data forward diffusion module 13 is used to input the target noise image into the target video generation model, extract the target noise image features through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector;

[0257] The feature encoding module 14 is configured to input the business text, business image, and target frame text into the target video generation model, perform text encoding on the target frame text using the target video generation model to obtain text encoding features, and perform multimodal feature extraction on the business text and business image to obtain image-text encoding features; the business image is a frame image that has been generated in the video to be generated; the business text is a continuous frame text among P continuous frame texts that is used to represent the business image, and the business text is the continuous frame text that is sorted before the target frame text;

[0258] The image prediction module 15 is used to denoise the target noisy image according to the target forward noise vector, text coding features and image-text coding features, obtain the target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located in the business image in the video to be generated.

[0259] In a possible implementation, when the data determination module 12 is used to obtain P noise images, the data determination module 12 is specifically used to perform the following operations:

[0260] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0261] Randomly obtain M noise data from a noise data pool, splice the M noise data to obtain a first spliced ​​noise map, perform Fourier transform on the first spliced ​​noise map to obtain a first frequency domain corresponding to the first spliced ​​noise map, and determine a frequency signal in the first frequency domain that is less than a low-frequency threshold as a global low-frequency signal; M is a positive integer less than S;

[0262] Obtain a first sliding step length, obtain P noise data groups from the S noise data using the first sliding step length, and splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M;

[0263] Performing Fourier transform on each of the P second spliced ​​noise images to obtain second frequency domains corresponding to the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to a low-frequency threshold as high-frequency signals;

[0264] The high-frequency signal in each second frequency domain is combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are inverse Fourier transformed to obtain P noise images.

[0265] In a possible implementation, when the data determination module 12 is used to obtain P noise images, the data determination module 12 is further used to perform the following operations:

[0266] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0267] Obtain a second sliding step length, obtain P noise data groups from the S noise data through the second sliding step length, and splice the noise data in the P noise data groups to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step length is less than M.

[0268] In one possible implementation, the feature encoding module 14 is configured to perform multimodal feature extraction on the business text and the business image. When obtaining the image-text encoding features, the feature encoding module 14 is specifically configured to perform the following operations:

[0269] Through the image encoding component in the target video generation model, feature extraction is performed on the business image to obtain image extraction features;

[0270] Through the text encoding component in the target video generation model, feature extraction is performed on the business text to obtain text extraction features;

[0271] The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are aligned to obtain the image-text encoding features.

[0272] In one possible implementation, the image prediction module 15 is configured to perform denoising on the target noisy image based on the target forward noise vector, the text encoding feature, and the image-text encoding feature. When obtaining the target predicted image corresponding to the target frame text, the image prediction module 15 is specifically configured to perform the following operations:

[0273] Generate fusion coding features based on text coding features and image-text coding features;

[0274] In the back diffusion network layer of the target video generation model, based on the fusion coding features, the target forward noise vector is back-diffused in T time steps to obtain the target prediction noise; T is a positive integer;

[0275] Determine the target latent vector based on the target prediction noise and the target forward noise vector;

[0276] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.

[0277] In one possible implementation, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; when the image prediction module 15 is used to generate a fused coding feature based on the text coding feature and the image-text coding feature, the image prediction module 15 is specifically used to perform the following operations:

[0278] Perform feature splicing on the text coding features and the image-text coding features to obtain spliced ​​coding features;

[0279] The concatenated coding features are dimensionally fused through the feature fusion network in the target video generation model to obtain fused coding features; the fused coding features have the same dimension as the target forward noise vector.

[0280] In a possible implementation, the number of business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the to-be-generated video; the image-text coding feature includes F image-text coding sub-features, and each image-text coding sub-feature is obtained by performing multimodal feature extraction based on a business text and the represented business image;

[0281] When the image prediction module 15 is used to generate a fusion coding feature based on the text coding feature and the image-text coding feature, the image prediction module 15 is specifically used to perform the following operations:

[0282] The unit features in the F image-text encoding sub-features are combined into a sequence to be matched;

[0283] Perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature, sort each unit feature based on the feature matching degree of each unit feature, and obtain a feature matching sequence;

[0284] The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer;

[0285] The text encoding features and target matching features are concatenated to obtain fused encoding features.

[0286] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module 15 is used to perform back diffusion processing on the target forward noise vector in T time steps based on the fused coding features to obtain the target prediction noise. The image prediction module 15 is specifically used to perform the following operations:

[0287] At time step T i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;

[0288] The time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.

[0289] In a possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network and a bypass attention sub-network; the image prediction module 15 is used to i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is , the image prediction module 15 is specifically configured to perform the following operations:

[0290] At time step T i In the fusion encoding feature and iterative noise vector, input to the time step T i The corresponding noise prediction network;

[0291] By time step T i The corresponding cross-attention sub-network in the noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the first attention processing result;

[0292] By time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the second attention processing result;

[0293] Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

[0294] In one possible implementation, the image prediction module 15 is configured to perform a first cross-attention process on the fused coding feature and the initial noise vector. When the first attention process result is obtained, the image prediction module 15 is specifically configured to perform the following operations:

[0295] The initial noise vector is used as the query vector in the crisscross attention function, the fused encoded features are used as the key vector in the crisscross attention function, and the fused encoded features are used as the value vector in the crisscross attention function;

[0296] The product of the query vector and the key vector is determined as the first fusion feature through the cross attention function;

[0297] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;

[0298] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the first attention processing result.

[0299] In a possible implementation, the data processing device 1700 further includes a video determination module 16, which is specifically configured to perform the following operations:

[0300] When the video to be generated contains P consecutive frame images corresponding to the text, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include the target prediction image;

[0301] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.

[0302] In a possible implementation, the frame images corresponding to the P consecutive frames of text in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; the video determination module 16 is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module 16 is specifically used to perform the following operations:

[0303] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;

[0304] Based on the frame image P j , frame image P j+1 And the optical flow prediction results are used to generate the frame image P j and frame image P j+1 The intermediate frame images between.

[0305] Further, see Figure 18 , Figure 18 This is a schematic diagram of a data processing device provided in an embodiment of the present application. Figure 2 The data processing device 1800 may include: a sample acquisition module 21 , a sample determination module 22 , a sample forward diffusion module 23 , a sample encoding module 24 , a sample reverse diffusion module 25 , and a parameter adjustment module 26 .

[0306] The sample acquisition module 21 is used to acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0307] A sample determination module 22 is configured to determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples;

[0308] The sample forward diffusion module 23 is used to extract features of the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain a potential noise feature, and perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector;

[0309] The sample encoding module 24 is configured to perform text encoding on the target text sample to obtain a first encoding feature, and perform multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is ranked before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample;

[0310] The sample reverse diffusion module 25 is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector;

[0311] The parameter adjustment module 26 is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate the target video indicated by the target text based on the target text.

[0312] In a possible implementation, when the sample acquisition module 21 is used to acquire a sample sequence, the sample acquisition module 21 is specifically used to perform the following operations:

[0313] Obtain a sample video, obtain N video frames based on the order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same subject object and object scene, and the display perspectives of the subject object in the N image samples are different from each other;

[0314] Generating text samples for N image samples respectively to obtain N text samples; the text samples are used to indicate specific content in the corresponding image samples;

[0315] Obtain N noise samples, combine the N image samples, N text samples, and N noise samples to obtain a sample sequence.

[0316] In a possible implementation, when the sample acquisition module 21 is used to acquire N noise samples, the sample acquisition module 21 is specifically used to perform the following operations:

[0317] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0318] Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine the frequency signal in the first sample frequency domain that is less than the sample low-frequency threshold as the sample global low-frequency signal; R is a positive integer less than S;

[0319] Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data using the first sample sliding step length, and concatenate the noise data in the N sample noise data groups to obtain N second sample noise maps; the number of noise data included in each sample noise data group is R;

[0320] Performing Fourier transform on each of the N second sample noise images to obtain second sample frequency domains corresponding to the N second sample noise images, and determining frequency signals in the N second sample frequency domains that are greater than or equal to a sample low-frequency threshold as sample high-frequency signals;

[0321] Each sample high-frequency signal in the second sample frequency domain is combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are inverse Fourier transformed to obtain N noise samples.

[0322] In a possible implementation, when the sample acquisition module 21 is used to acquire N noise samples, the sample acquisition module 21 is further used to perform the following operations:

[0323] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0324] Obtain a second sample sliding step length, obtain N noise data groups from the S noise data through the second sample sliding step length, and splice the noise data in the N noise data groups to obtain N noise samples; the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step length is less than R.

[0325] In one possible implementation, the initial video generation model includes an initial back diffusion network layer, which includes a cross-attention subnetwork and an initial bypass attention subnetwork. The sample back diffusion module 25 is used to perform back diffusion processing on the sample forward noise vector, the first encoding feature, and the second encoding feature. When the predicted noise sample vector is obtained, the sample back diffusion module 25 is specifically used to perform the following operations:

[0326] generating a target coding feature based on the first coding feature and the second coding feature;

[0327] Through the cross-attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the first sample attention result;

[0328] Through the initial bypass attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the second sample attention result;

[0329] Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

[0330] In one possible implementation, the initial video generation model further includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature. The initial feature fusion network is used to generate a target coding feature based on the first coding feature and the second coding feature. The parameter adjustment module 26 is used to adjust the parameters of the initial video generation model based on N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples. When the target video generation model is obtained, the parameter adjustment module 26 is specifically used to perform the following operations:

[0331] Generate a model loss value based on N noise samples and the predicted noise sample vectors predicted by the N noise samples;

[0332] Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain the target video generation model.

[0333] See Figure 19 , Figure 19 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 19 As shown, the computer device 1900 in the embodiment of the present application may include: a processor 1901, a network interface 1904 and a memory 1905. In addition, the above-mentioned computer device 1900 may also include: a user interface 1903, and at least one communication bus 1902. Among them, the communication bus 1902 is used to realize the connection and communication between these components. Among them, the user interface 1903 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1903 may also include a standard wired interface and a wireless interface. The network interface 1904 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1905 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1905 may optionally also be at least one storage device located away from the aforementioned processor 1901. As Figure 19 As shown, the memory 1905 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0334] The network interface 1904 may provide a network communication element; the user interface 1903 may be mainly used to provide an input interface for the user; and the processor 1901 may be used to call a computer program stored in the memory 1905 to execute the steps in each embodiment of the present application.

[0335] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program is suitable for being loaded and executed by the processor Figure 3 、 Figure 4 、 Figure 10 、 Figure 13 or Figure 15 For details on the methods provided in each step, please refer to the Figure 3 、 Figure 4 、 Figure 10 、 Figure 13 or Figure 15 The implementation methods provided in each step will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the computer program can be deployed to be executed on one computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0336] The computer-readable storage medium may be the device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0337] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs Figure 3 、 Figure 4 、 Figure 10 、 Figure 13 or Figure 15 The methods provided in the various optional methods are therefore not described in detail here.

[0338] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0339] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0340] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0341] The methods and related devices provided in the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer programs. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the computer program executed by the processor of the computer or other programmable device generates a program for implementing the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1The computer programs can also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the computer program stored in the computer-readable memory produces a manufactured product including the program device, or is transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (for example, coaxial cable, optical fiber, digital line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.). The program device is implemented in the process Figure 1 Schematic diagram of one or more processes and / or structures Figure 1 These computer programs can also be loaded onto computers or other programmable devices so that a series of operating steps are executed on the computer or other programmable devices to produce computer-implemented processing, thereby providing a computer program executed on the computer or other programmable device for implementing the functions specified in the process. Figure 1 The flow or flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0342] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0343] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0344] The above disclosure is only a preferred embodiment of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer; Acquire P noise images, acquire target frame texts traversed in order from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images; Inputting the target noise image into a target video generation model, performing feature extraction on the target noise image through the target video generation model to obtain target noise features, and performing forward diffusion processing on the target noise features to obtain a target forward noise vector; Inputting the business text, the business image, and the target frame text into the target video generation model, performing text encoding on the target frame text using the target video generation model to obtain text encoding features, and performing multimodal feature extraction on the business text and the business image to obtain image-text encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text among P continuous frame texts used to represent the business image, and the business text is a continuous frame text that is sorted before the target frame text; The target noise image is denoised according to the target forward noise vector, the text encoding feature and the image-text encoding feature to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image of the business image in the video to be generated.

2. The method according to claim 1, characterized in that The acquiring P noise images includes: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Randomly obtaining M noise data from the noise data pool, splicing the M noise data to obtain a first spliced ​​noise image, performing Fourier transform on the first spliced ​​noise image to obtain a first frequency domain corresponding to the first spliced ​​noise image, and determining a frequency signal in the first frequency domain that is less than a low-frequency threshold as a global low-frequency signal; M is a positive integer less than S; Obtaining a first sliding step length, obtaining P noise data groups from the S noise data using the first sliding step length, and splicing the noise data in the P noise data groups to obtain P second spliced ​​noise maps; the number of noise data included in each noise data group is M; Performing Fourier transform on each of the P second spliced ​​noise images to obtain second frequency domains corresponding to each of the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to the low-frequency threshold as high-frequency signals; The high-frequency signal in each second frequency domain is respectively combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are respectively subjected to inverse Fourier transform to obtain P noise images.

3. The method according to claim 1, characterized in that The acquiring P noise images includes: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Obtain a second sliding step size, obtain P noise data groups from the S noise data using the second sliding step size, and splice the noise data in the P noise data groups to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step size is less than M.

4. The method according to claim 1, wherein The performing multimodal feature extraction on the business text and the business image to obtain image-text coding features includes: Performing feature extraction on the business image through the image encoding component in the target video generation model to obtain image extraction features; Performing feature extraction on the business text through the text encoding component in the target video generation model to obtain text extraction features; The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are feature aligned to obtain image-text coding features.

5. The method according to claim 1, wherein The denoising process is performed on the target noise image according to the target forward noise vector, the text encoding feature, and the image-text encoding feature to obtain a target predicted image corresponding to the target frame text, including: Generate a fusion coding feature based on the text coding feature and the image-text coding feature; In the back diffusion network layer in the target video generation model, based on the fused coding features, the target forward noise vector is back diffused in T time steps to obtain target prediction noise; T is a positive integer; determining a target latent vector based on the target prediction noise and the target forward noise vector; The target latent vector is reconstructed by a decoder in the target video generation model to obtain a target predicted image corresponding to the target frame text.

6. The method according to claim 5, characterized in that The business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; generating the fused coding feature according to the text coding feature and the image-text coding feature includes: Performing feature splicing on the text coding feature and the image-text coding feature to obtain a spliced ​​coding feature; The spliced ​​coding features are dimensionally fused through a feature fusion network in the target video generation model to obtain fused coding features; the fused coding features have the same dimension as the target forward noise vector.

7. The method according to claim 5, characterized in that The number of the business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the video to be generated; the image-text coding feature includes F image-text coding sub-features, and each image-text coding sub-feature is obtained by performing multimodal feature extraction based on a business text and the represented business image; The generating of a fusion coding feature according to the text coding feature and the image-text coding feature includes: Combining the unit features in the F image-text encoding sub-features into a sequence to be matched; Performing feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain a feature matching degree corresponding to each unit feature, and sorting each unit feature based on the feature matching degree of each unit feature to obtain a feature matching sequence; The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer; The text encoding feature and the target matching feature are concatenated to obtain a fused encoding feature.

8. The method according to claim 5, characterized in that The T time steps include time step T i , i is a positive integer less than or equal to T; the target forward noise vector is subjected to reverse diffusion processing in T time steps based on the fused coding feature to obtain target prediction noise, including: At the time step T i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step of the T time steps, then the time step T i The corresponding iterative noise vector is the target forward noise vector; The time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector at the time step T i+1 In the example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; the time step T i+1 is the time step T in the T time steps i The next time step.

9. The method according to claim 8, characterized in that The back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network and a bypass attention sub-network; at the time step T i In the example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector includes: At the time step T i In the process, the fused coding feature and the iterative noise vector are input to the time step T i The corresponding noise prediction network; By the time step T i The cross attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding feature and the iterative noise vector to obtain a first attention processing result; By the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding feature and the iterative noise vector to obtain a second attention processing result; Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

10. The method according to claim 9, characterized in that The performing a first cross attention process on the fused coding feature and the initial noise vector to obtain a first attention processing result includes: Using the initial noise vector as a query vector in a crisscross attention function, using the fused encoded features as a key vector in the crisscross attention function, and using the fused encoded features as a value vector in the crisscross attention function; Determining a first fused feature by multiplying the query vector and the key vector through the cross attention function; Obtaining the number of dimensions corresponding to the fused coding feature, and determining the product of the first fused feature and the inverse of the number of dimensions as a second fused feature; Based on the activation sub-function in the cross attention function, the second fused feature is converted into a first activation feature, and the product of the first activation feature and the value vector is determined as a first attention processing result.

11. The method according to claim 1, wherein Also includes: When the video to be generated includes frame images corresponding to the P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include the target predicted image; The video to be generated, which includes the frame images corresponding to the P consecutive frame texts and the intermediate frame images, is determined as a business video matching the target text.

12. The method according to claim 11, characterized in that The frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; performing image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images includes: Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain an optical flow prediction result; the frame image P j+1 The frame image P in the video to be generated j The next frame image, the frame image P j+2 The frame image P in the video to be generated j+1 The next frame image, the frame image P j-1 is the frame image P in the video to be generated j The previous frame image; Based on the frame image P j , the frame image P j+1 and the optical flow prediction result to generate the frame image P j and frame image P j+1 The intermediate frame images between.

13. A data processing method, characterized in that: The method comprises: Obtain a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer; Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples; Performing feature extraction on the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain a latent noise feature, and performing forward diffusion processing on the latent noise feature to obtain a sample forward noise vector; Performing text encoding on the target text sample to obtain a first encoding feature, and performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample; Performing a back-diffusion process on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector; According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

14. The method according to claim 13, characterized in that The acquiring of the sample sequence comprises: Obtain a sample video, obtain N video frames based on a sorting order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other; Generating text samples for the N image samples respectively to obtain N text samples; the text samples are used to indicate specific content in the corresponding image samples; Obtain N noise samples, and combine the N image samples, the N text samples, and the N noise samples to obtain a sample sequence.

15. The method according to claim 14, characterized in that The obtaining of N noise samples includes: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine a frequency signal in the first sample frequency domain that is less than a sample low-frequency threshold as a sample global low-frequency signal; R is a positive integer less than S; Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data using the first sample sliding step length, and concatenate the noise data in the N sample noise data groups to obtain N second sample noise maps; the number of noise data included in each sample noise data group is R; Performing Fourier transform on each of the N second sample noise images to obtain second sample frequency domains corresponding to each of the N second sample noise images, and determining a frequency signal in each of the N second sample frequency domains that is greater than or equal to the sample low-frequency threshold as a sample high-frequency signal; Each sample high-frequency signal in the second sample frequency domain is combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are inverse Fourier transformed to obtain N noise samples.

16. The method according to claim 14, characterized in that The obtaining of N noise samples includes: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Obtain a second sample sliding step size, obtain N noise data groups from the S noise data using the second sample sliding step size, and concatenate the noise data in the N noise data groups to obtain N noise samples; the number of noise data included in each noise data group is R, where R is a positive integer, and the second sample sliding step size is less than R.

17. The method according to claim 13, wherein The initial video generation model includes an initial back-diffusion network layer, which includes a cross-attention sub-network and an initial bypass attention sub-network; performing back-diffusion processing on the sample forward noise vector, the first encoding feature, and the second encoding feature to obtain a predicted noise sample vector, including: generating a target coding feature according to the first coding feature and the second coding feature; Performing cross-attention processing on the sample forward noise vector and the target encoding feature through the cross-attention sub-network of the initial back-diffusion network layer to obtain a first sample attention result; Performing cross-attention processing on the sample forward noise vector and the target encoding feature through the initial bypass attention subnetwork of the initial back-diffusion network layer to obtain a second sample attention result; Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

18. The method according to claim 17, characterized in that The initial video generation model further includes an initial multimodal feature extraction network and an initial feature fusion network, wherein the initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature; and the initial feature fusion network is used to generate a target coding feature based on the first coding feature and the second coding feature. The step of adjusting parameters of an initial video generation model according to the N noise samples and predicted noise sample vectors respectively predicted by the N noise samples to obtain a target video generation model includes: Generate a model loss value according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples; Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain a target video generation model.

19. A data processing device, characterized in that: The device comprises: The data acquisition module is used to acquire the target text and split the target text to obtain P consecutive frame texts in the order of video frames; P is a positive integer; a data determination module, configured to obtain P noise images, obtain a target frame text traversed in sequence from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images; a data forward diffusion module, configured to input the target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector; A feature encoding module is configured to input the business text, the business image, and the target frame text into the target video generation model, perform text encoding on the target frame text using the target video generation model to obtain text encoding features, and perform multimodal feature extraction on the business text and the business image to obtain image-text encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text among P continuous frame texts used to represent the business image, and the business text is a continuous frame text that is sorted before the target frame text; An image prediction module is used to denoise the target noisy image according to the target forward noise vector, the text encoding feature and the image-text encoding feature to obtain a target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image of the business image in the video to be generated.

20. A data processing device, characterized in that: The device comprises: A sample acquisition module is used to acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer; a sample determination module, configured to determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples; a sample forward diffusion module, configured to extract features from a target image sample and a target noise sample corresponding to the target image sample among the N noise samples to obtain a potential noise feature, and perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; a sample encoding module, configured to perform text encoding on a target text sample to obtain a first encoding feature, and perform multimodal feature extraction on a first image sample and a first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample; a sample back diffusion module, configured to perform back diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector; A parameter adjustment module is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

Images