Data processing method and device

Abstract

The embodiment of the invention discloses a data processing method and device, and relates to the technical field of computers, and the method comprises the steps: obtaining a target text, and obtaining P continuous frame texts through the target text; obtaining P noise images, obtaining a target frame text from the P continuous frame texts, and determining a target noise image from the P noise images; processing the target noise image through a target video generation model to obtain a target forward noise vector; and processing the business text, the business image and the target frame text through a target video generation model to obtain a fused coding feature, and performing denoising processing on the target noise image according to the target forward noise vector and the fused coding feature to obtain a target prediction image corresponding to the target frame text, and taking the target prediction image as a next frame image located in the business image in the video to be generated. By adopting the method and the device, the overall tone consistency of the text video can be improved, and the quality of the text video is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the continuous progress of generative adversarial networks and diffusion models, the field of image and video generation has ushered in a major breakthrough. Text-to-video technology can generate corresponding images only through text descriptions. The core of this technology is to combine natural language processing with computer vision technology, and use trained models to convert text information into visual content. Based on image generation technology, text-to-video is a product that further expands this capability. Text-to-video requires not only that the model can generate static images, but also that it can generate continuous dynamic images based on text descriptions. At present, with the help of the open source stable diffusion model (SD model), the first frame of the text can be generated under specific control parameters, and then the second frame of the image can be generated under some of the same control parameters until all the images required for the video sequence are generated and the video is generated. However, when the generated text input by the SD model changes with respect to the previous frame of text, it is easy to cause the generated foreground and background to be obviously inconsistent with the previous frame when the next frame is generated, resulting in poor generation effects and low quality of the text-to-video. Summary of the invention

[0003] The embodiments of the present application provide a data processing method and device, which can improve the overall tone consistency of the Vincent video and ensure the quality of the Vincent video.

[0004] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0005] Obtain the target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer;

[0006] Obtain P noise images, obtain the target frame text traversed in order from the P consecutive frame texts, and determine the target noise image associated with the target frame text from the P noise images;

[0007] The target noise image is input into the target video generation model, and the target noise image is subjected to feature extraction through the target video generation model to obtain the target noise feature, and the target noise feature is subjected to forward diffusion processing to obtain the target forward noise vector;

[0008] The business text, business image and target frame text are input into the target video generation model, the target frame text is encoded by the target video generation model to obtain text encoding features, and multimodal features are extracted from the business text and business image to obtain image-text encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text;

[0009] The target noise image is denoised according to the target forward noise vector, text encoding features and image-text encoding features to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located in the business image in the video to be generated.

[0010] On the one hand, an embodiment of the present application provides another data processing method, the method comprising:

[0011] Obtain a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0012] Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples;

[0013] Extract features of the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain potential noise features, perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector;

[0014] Performing text encoding on the target text sample to obtain a first encoding feature, and performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample;

[0015] Performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector;

[0016] According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0017] An embodiment of the present application provides a data processing device, the device comprising:

[0018] The data acquisition module is used to acquire the target text, split the target text, and obtain P consecutive frame texts with the video frame sorting order; P is a positive integer;

[0019] A data determination module is used to obtain P noise images, obtain the target frame text traversed in order from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images;

[0020] A data forward diffusion module is used to input a target noise image into a target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector;

[0021] A feature encoding module is used to input the business text, business image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image to obtain image-text encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text;

[0022] The image prediction module is used to denoise the target noisy image according to the target forward noise vector, text encoding features and image-text encoding features, obtain the target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located in the business image in the video to be generated.

[0023] In a possible implementation, when the data determination module is used to obtain P noise images, the data determination module is specifically used to perform the following operations:

[0024] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0025] Randomly obtain M noise data from the noise data pool, splice the M noise data to obtain a first spliced ​​noise image, perform Fourier transform on the first spliced ​​noise image to obtain a first frequency domain corresponding to the first spliced ​​noise image, and determine a frequency signal less than a low-frequency threshold in the first frequency domain as a global low-frequency signal; M is a positive integer less than S;

[0026] Obtain a first sliding step length, obtain P noise data groups from the S noise data through the first sliding step length, and respectively splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M;

[0027] Performing Fourier transform on the P second spliced ​​noise images respectively to obtain second frequency domains respectively corresponding to the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to a low-frequency threshold as high-frequency signals;

[0028] The high-frequency signals in each second frequency domain are respectively combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are respectively inverse Fourier transformed to obtain P noise images.

[0029] In a possible implementation manner, when the data determination module is used to obtain P noise images, the data determination module is specifically further used to perform the following operations:

[0030] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0031] A second sliding step is obtained, and P noise data groups are obtained from the S noise data through the second sliding step, and the noise data in the P noise data groups are respectively spliced ​​to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step is less than M.

[0032] In a possible implementation, the feature encoding module is used to extract multimodal features from the business text and the business image. When the image-text encoding features are obtained, the feature encoding module is specifically used to perform the following operations:

[0033] Through the image encoding component in the target video generation model, feature extraction is performed on the business image to obtain image extraction features;

[0034] Through the text encoding component in the target video generation model, feature extraction is performed on the business text to obtain text extraction features;

[0035] The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are aligned to obtain the image-text encoding features.

[0036] In a possible implementation, the image prediction module is used to perform denoising on the target noise image according to the target forward noise vector, the text encoding feature and the image-text encoding feature, and when the target predicted image corresponding to the target frame text is obtained, the image prediction module is specifically used to perform the following operations:

[0037] Generate fused coding features according to text coding features and image-text coding features;

[0038] In the reverse diffusion network layer in the target video generation model, based on the fusion coding features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer;

[0039] Determine a target latent vector based on the target prediction noise and the target forward noise vector;

[0040] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.

[0041] In a possible implementation, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; when the image prediction module is used to generate a fused coding feature according to the text coding feature and the image-text coding feature, the image prediction module is specifically used to perform the following operations:

[0042] Perform feature splicing on the text coding feature and the image-text coding feature to obtain the spliced ​​coding feature;

[0043] The concatenated coding features are dimensionally fused through a feature fusion network in a target video generation model to obtain a fused coding feature; the fused coding feature has the same dimension as the target forward noise vector.

[0044] In a possible implementation, the number of business texts and business images is F, F is a positive integer, and the F business images are all frame images generated in the video to be generated; the image-text coding feature includes F image-text coding sub-features, and one image-text coding sub-feature is obtained by performing multimodal feature extraction based on one business text and the represented business image;

[0045] When the image prediction module is used to generate fused coding features according to the text coding features and the image-text coding features, the image prediction module is specifically used to perform the following operations:

[0046] The unit features in the F image-text encoding sub-features are combined into a sequence to be matched;

[0047] Perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature, sort each unit feature based on the feature matching degree of each unit feature, and obtain a feature matching sequence;

[0048] The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer;

[0049] The text encoding features and target matching features are concatenated to obtain fused encoding features.

[0050] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module is used to perform reverse diffusion processing on the target forward noise vector in T time steps based on the fusion coding features to obtain the target prediction noise. The image prediction module is specifically used to perform the following operations:

[0051] At time step T i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;

[0052] Set the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the above example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.

[0053] In a possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network and a bypass attention sub-network; the image prediction module is used to i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is , the image prediction module is specifically used to perform the following operations:

[0054] At time step T i In the example, the fused encoded features and iterative noise vector are input to the time step T i The corresponding noise prediction network;

[0055] Through the time step T i The cross attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding features and the iterative noise vector to obtain the first attention processing result;

[0056] Through the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the second attention processing result;

[0057] Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

[0058] In a possible implementation, the image prediction module is used to perform a first cross attention process on the fused coding feature and the initial noise vector. When the first attention process result is obtained, the image prediction module is specifically used to perform the following operations:

[0059] The initial noise vector is used as the query vector in the cross-attention function, the fused encoded features are used as the key vector in the cross-attention function, and the fused encoded features are used as the value vector in the cross-attention function;

[0060] The product of the query vector and the key vector is determined as the first fused feature through the cross attention function;

[0061] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;

[0062] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the first attention processing result.

[0063] In a possible implementation, the data processing device further includes a video determination module, and the video determination module is specifically configured to perform the following operations:

[0064] When the video to be generated contains frame images corresponding to P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include a target prediction image;

[0065] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.

[0066] In a possible implementation, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image Pj+1 , j is a positive integer less than P; the video determination module is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module is specifically used to perform the following operations:

[0067] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;

[0068] Based on the frame image P j , frame image P j+1 And the optical flow prediction results, generate the frame image P j and frame image P j+1 The intermediate frame images between.

[0069] On the one hand, an embodiment of the present application provides another data processing device, which includes:

[0070] The sample acquisition module is used to acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0071] A sample determination module, used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples;

[0072] A sample forward diffusion module is used to extract features of a target image sample and a target noise sample corresponding to the target image sample among N noise samples to obtain a potential noise feature, and to perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector;

[0073] A sample encoding module is used to perform text encoding on a target text sample to obtain a first encoding feature, and to perform multimodal feature extraction on a first image sample and a first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample;

[0074] A sample reverse diffusion module is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector;

[0075] The parameter adjustment module is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0076] In a possible implementation, when the sample acquisition module is used to acquire a sample sequence, the sample acquisition module is specifically used to perform the following operations:

[0077] Obtain a sample video, obtain N video frames based on the sorting order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other;

[0078] Generating text samples for N image samples respectively to obtain N text samples; the text samples are used to indicate specific contents in the corresponding image samples;

[0079] Obtain N noise samples, combine the N image samples, the N text samples, and the N noise samples to obtain a sample sequence.

[0080] In a possible implementation, when the sample acquisition module is used to acquire N noise samples, the sample acquisition module is specifically used to perform the following operations:

[0081] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0082] Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine a frequency signal in the first sample frequency domain that is less than the sample low-frequency threshold as a sample global low-frequency signal; R is a positive integer less than S;

[0083] Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data through the first sample sliding step length, and respectively splice the noise data in the N sample noise data groups to obtain N second sample noise images; the number of noise data included in each sample noise data group is R;

[0084] Performing Fourier transform on the N second sample noise images respectively to obtain second sample frequency domains corresponding to the N second sample noise images respectively, and determining frequency signals in the N second sample frequency domains that are greater than or equal to the sample low-frequency threshold as sample high-frequency signals respectively;

[0085] The sample high-frequency signal in each second sample frequency domain is respectively combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are respectively inverse Fourier transformed to obtain N noise samples.

[0086] In a possible implementation, when the sample acquisition module is used to acquire N noise samples, the sample acquisition module is further used to perform the following operations:

[0087] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0088] A second sample sliding step is obtained, and N noise data groups are obtained from the S noise data through the second sample sliding step. The noise data in the N noise data groups are respectively spliced ​​to obtain N noise samples; the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step is less than R.

[0089] In a possible implementation, the initial video generation model includes an initial back diffusion network layer, which includes a cross attention sub-network and an initial bypass attention sub-network; the sample back diffusion module is used to perform back diffusion processing on the sample forward noise vector, the first encoding feature, and the second encoding feature, and when the predicted noise sample vector is obtained, the sample back diffusion module is specifically used to perform the following operations:

[0090] generating a target coding feature according to the first coding feature and the second coding feature;

[0091] Through the cross-attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the first sample attention result;

[0092] Through the initial bypass attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the second sample attention result;

[0093] Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

[0094] In a possible implementation, the initial video generation model also includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature. The initial feature fusion network is used to generate a target coding feature based on the first coding feature and the second coding feature. The parameter adjustment module is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples. When the target video generation model is obtained, the parameter adjustment module is specifically used to perform the following operations:

[0095] Generate a model loss value according to N noise samples and predicted noise sample vectors respectively predicted by the N noise samples;

[0096] Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain the target video generation model.

[0097] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;

[0098] The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that a computer device including the processor executes the method in one aspect of an embodiment of the present application.

[0099] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method in one aspect of the embodiment of the present application.

[0100] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional ways in one aspect of the embodiment of the present application. In other words, when the computer program is executed by the processor, the method provided in various optional ways in one aspect of the embodiment of the present application is implemented.

[0101] In an embodiment of the present application, a target text is obtained, and the target text is split to obtain P continuous frame texts with a video frame sorting order; P is a positive integer. P noise images are obtained, and the target frame text traversed in order is obtained from the P continuous frame texts, and the target noise image associated with the target frame text is determined from the P noise images. The target noise image is input into the target video generation model, and the target noise image is feature extracted by the target video generation model to obtain the target noise feature, and the target noise feature is forward diffused to obtain the target forward noise vector. The business text, business image and target frame text are input into the target video generation model, and the target frame text is text encoded by the target video generation model to obtain the text encoding feature, and the business text and business image are multimodal feature extracted to obtain the graphic encoding feature; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image in the P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text. The feature extraction method can be fully utilized to extract the features of historical images (business images) and historical texts (business texts); the target noise image is denoised according to the target forward noise vector, text encoding features and image-text encoding features to obtain the target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located in the business image in the video to be generated. The target video generation model is constrained and guided by the business text and business image together with the latest frame text (target frame text), and the target prediction image corresponding to the target frame text is generated by the target noise image, so as to ensure that the contents (subject object and object scene, etc.) of the generated target prediction image and the historical image (i.e., the latest frame in the video to be generated, i.e., the target prediction image and the historical frame) can be kept consistent, avoiding the excessive difference between the generated latest frame and the historical frame due to the change of the latest frame text, thereby improving the similarity between the target prediction image and the historical image, and thus improving the coherence and coordination of the video to be generated based on the target text; at the same time, the constraints of historical text and historical image on the generation of the target prediction image by the target video generation model can prompt the generation style or content of the target prediction image to be biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby improving the overall tone consistency of the video, and further ensuring the quality of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0103] Figure 1 This is a network interaction architecture diagram provided by an embodiment of the present application;

[0104] Figure 2 It is a scenario schematic diagram of a data processing method provided in an embodiment of the present application;

[0105] Figure 3 This is a data processing method provided by the embodiment of the present application. Figure 1 ;

[0106] Figure 4 This is a data processing method provided by the embodiment of the present application. Figure 2 ;

[0107] Figure 5 This is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application. Figure 1 ;

[0108] Figure 6 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 1 ;

[0109] Figure 7 It is a schematic diagram of a downsampling network layer structure provided in an embodiment of the present application;

[0110] Figure 8 is a schematic diagram of a transformation processing layer structure provided in an embodiment of the present application;

[0111] Fig. 9 is a schematic diagram of a transformation processing unit structure provided in an embodiment of the present application;

[0112] Fig.10 This is a data processing method provided by the embodiment of the present application. Figure 3 ;

[0113] Fig.11 This is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application. Figure 2 ;

[0114] Fig.12 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 2 ;

[0115] Fig.13 This is a data processing method provided by the embodiment of the present application. Figure 4 ;

[0116] Fig.14 This is a schematic diagram of an initial video generation model structure provided in an embodiment of the present application. Figure 1 ;

[0117] Fig.15This is a data processing method provided by the embodiment of the present application. Figure 5 ;

[0118] Fig.16 This is a schematic diagram of an initial video generation model structure provided in an embodiment of the present application. Figure 2 ;

[0119] Fig.17 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0120] Fig.18 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0121] Fig.19 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0122] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0123] Among them, if it is necessary to collect object (such as user, etc.) data in this application, a prompt interface or pop-up window will be displayed before or during the collection. The prompt interface or pop-up window is used to prompt the user that certain data is currently being collected. Only after the user confirms the prompt interface or pop-up window, the relevant steps of data acquisition will be started, otherwise the process will end. Moreover, the acquired user data will be used in reasonable and legal scenarios or purposes. Optionally, in some scenarios where user data needs to be used but has not been authorized by the user, authorization can be requested from the user, and the user data can be used when the authorization is passed.

[0124] It is understandable that in the specific implementation of this application, the user data involved, when the following embodiments of this application are applied to specific products or technologies, needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.

[0125] In the examples of this application, see Figure 1 , Figure 1 is a network interaction architecture diagram provided by an embodiment of the present application, such as Figure 1As shown, the network interaction architecture diagram may include a business server 101 and a terminal device cluster, and the terminal device cluster may include terminal device 102a, terminal device 102b, terminal device 102c, ..., terminal device 102n, wherein there may be a communication connection between the terminal device clusters, for example, there is a communication connection between terminal device 102a and terminal device 102b, and there is a communication connection between terminal device 102a and terminal device 102c. At the same time, any terminal device in the terminal device cluster may have a communication connection with the business server 101, for example, there is a communication connection between terminal device 102a and business server 101, wherein the above communication connection does not limit the connection method, and may be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and this application does not make any restrictions here.

[0126] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown in the figure can be installed with an application client having a data processing function. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1 The business servers 101 shown in the figure interact with each other so that the business server 101 can receive business data from each terminal device. Among them, the application client can be a social application, instant messaging application, live broadcast application, short video application, video application, music application, shopping application, novel application, browser, etc., which has the function of displaying text, images, audio, video and other data information. Among them, the application client can be an independent client or an embedded sub-client (for example, application applet, browser web page access, etc.) integrated in a client (for example, instant messaging client, social client, video client, etc.), which is not limited here.

[0127] like Figure 1As shown, any terminal device in the terminal device cluster can send a video generation request containing a target text to the business server 101. Taking the terminal device 102a as an example, the terminal device 102a can send the target text to the business server 101 through the video generation request. The target text is the description text of the video to be generated indicated by the video generation request. The business server 101 can receive the video generation request sent by the terminal device 102a, obtain the target text from the video generation request, expand the target text based on the semantic information of the target text, and split the expanded target text to obtain P continuous frame texts with a video frame sorting order; P is a positive integer. The P continuous frame texts are respectively used to generate P frame images in the video to be generated, and are respectively used to characterize the description texts of the P frame images in the video to be generated. It can be understood that in order to maintain the coherence of the video to be generated, the difference between two adjacent continuous frame texts in the P continuous frame texts is small, that is, the text description content or style of the two continuous frame texts is similar. For example, the target text can be represented as "a cat is chasing a butterfly on the grass, then suddenly jumps onto a stone, and finally sits on the stone and licks its paws.", where two adjacent continuous frame texts can be represented as "the cat starts to lick its front paw with its tongue" and "the cat switches to the other paw and continues to lick it with its tongue" respectively.

[0128] The business server 101 can obtain P noise images, determine the target frame text from the P continuous frame texts by traversing in order, and determine the target noise image associated with the target frame text from the P noise images. The business server 101 can extract features of the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features, and obtain a target forward noise vector. The business server 101 can input the business text, business image, and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image, and obtain graphic encoding features. Among them, the business image belongs to the frame image that has been generated in the video to be generated, that is, the historical frame image; the business text is the continuous frame text used to characterize the business image in the P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0129] The business server 101 can perform denoising on the target noise image according to the target forward noise vector, text encoding features and image-text encoding features, obtain the target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated. When the target prediction image corresponding to the last continuous frame text is generated, the business video indicated by the target text can be obtained, and the business video includes frame images corresponding to P continuous frame texts, that is, P frame images. The business server 101 can send the business video to the terminal device 102a so that the terminal device 102a can display the business video.

[0130] The embodiment of the present application jointly constrains and guides the target video generation model through business images, business texts, and target frame texts to generate a target prediction image for the target frame text, so that the features of the business images and business texts can play a role in the process of generating the target prediction image, ensuring that the generated target prediction image and the content (subject object and object scene, etc.) of the historical image (i.e., the newly generated frame and historical frame of the video to be generated) can be consistent, avoiding excessive differences in the generated images due to changes in the latest frame text (target frame text), improving the similarity between the target prediction image and the historical image, and thus improving the coherence and coordination of the video to be generated based on the target text; at the same time, the constraints of historical texts and historical images on the generation of the target prediction image by the target video generation model can prompt the generation style or content of the target prediction image to be biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby improving the overall tone consistency of the video and further ensuring the quality of the video.

[0131] It is understandable that the terminal device mentioned in the embodiments of the present application may also be a computer device, and the computer device in the embodiments of the present application includes but is not limited to a terminal device or a server. In other words, the computer device may be a server or a terminal device, or it may be a system composed of a server and a terminal device. Among them, the terminal device mentioned above may be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. As Figure 1As shown in , the terminal device may be a mobile phone (as shown in terminal device 102a), a desktop computer (as shown in terminal device 102b), a tablet computer (as shown in terminal device 102c) or a laptop computer (as shown in terminal device 102n), etc. Figure 1 Only some of the devices are listed. The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0132] Optionally, the data involved in the embodiments of the present application can be stored in a computer device, or the data can be stored based on cloud storage technology or a blockchain network, which is not limited here.

[0133] For details, see Figure 2 , Figure 2 Schematic diagram of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the service server can obtain the target text in the video generation request sent by the terminal device, wherein the service server can be Figure 1 The service server 101 shown, the terminal device can be Figure 1 Any terminal device in the terminal device cluster shown in the figure, here taking the terminal device 102a as an example. The user corresponding to the terminal device 102a inputs the video description (target text) and initiates a video generation request. The service server 101 can obtain the target text in the video generation request sent by the terminal device 102a, split the target text into P continuous frame texts (including continuous frame text C1, continuous frame text C2, ..., continuous frame text C P ). The service server 101 may obtain P noise images, obtain the target frame text traversed in order from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images. For example, when the target frame text is the first continuous frame text (continuous frame text C1) among the P continuous frame texts, the target noise image may be the first noise image among the P noise images. Similarly, when the target frame text is the last continuous frame text (continuous frame text C1) among the P continuous frame texts, the target noise image may be the first noise image among the P noise images. P ), the target noise image may be the last noise image among the P noise images, that is, the Pth noise image.

[0134] The business server 101 can input the target noise image into the target video generation model, extract the target noise image through the target video generation model to obtain the target noise feature, and perform forward diffusion processing on the target noise feature to obtain the target forward noise vector. Among them, the target video generation model can be obtained by modifying the SD model, and the business server 101 can perform forward diffusion processing on the target noise feature through the forward diffusion network layer of the target video generation model to obtain the target forward noise vector. The forward diffusion network layer can be the forward diffusion network layer in the SD model. The business server 101 can input business text, business image and target frame text into the target video generation model, and perform text encoding on the target frame text through the target video generation model to obtain text encoding features; perform multimodal feature extraction on the business text and business image to obtain graphic encoding features. Among them, the business image belongs to the frame image that has been generated in the video to be generated, that is, the continuous frame text (historical text) that is sorted before the target frame text, and the target prediction image corresponding to each historical text generated by the target video generation model; the business text is the continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text. For example, when the target frame text is the third continuous frame text (i.e., continuous frame text C3) among the P continuous frame texts, the business text can be the continuous frame text sorted before the target frame text, that is, including the first continuous frame text (i.e., continuous frame text C1) and the second continuous frame text (i.e., continuous frame text C2) among the P continuous frame texts. It should be noted that when the target frame text is the first continuous frame text (i.e., continuous frame text C1) among the P continuous frame texts, the business image and business text input into the target video generation model by the business server 101 can be null value input, that is, the input business image is a blank null value image, and the input business text is a blank null value text.

[0135] Furthermore, the business server 101 can generate a model for the target video, perform denoising on the target noise image according to the target forward noise vector, text encoding features, and image-text encoding features, obtain a target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated. When the business server 101 obtains target prediction images corresponding to P consecutive frames of text, that is, P target prediction images (including target prediction image B1, target prediction image B2, ..., target prediction image B P ), the final video to be generated can be obtained. The final video to be generated includes P frame images, the target prediction image B1 is the target prediction image corresponding to the continuous frame text C1, the target prediction image B2 is the target prediction image corresponding to the continuous frame text C2, and the target prediction image B P For continuous frame text CP The corresponding target predicted image. Optionally, the business server 101 can further perform frame insertion processing on the final video to be generated, further improve the inter-frame coherence of the video to be generated, and obtain a business video. That is, the business server 101 inserts one or more intermediate frame images into each two adjacent frame images of the P frame images of the final video to be generated, so that the transition between the two adjacent frame images is more natural, and the obtained business video is more coherent. Among them, the business video includes the P frame images in the video to be generated, and the intermediate frame images inserted between each two adjacent frame images. The business server 101 can send the business video to the terminal device 102a, so that the terminal device 102a can display the business video to the user at the front end.

[0136] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business image and the business text play a role in the process of generating the target prediction image, which makes the generated target prediction image and the historical image (i.e., the newly generated frame and the historical frame in the video to be generated) The content (subject object and object scene, etc.) can be kept consistent, avoiding the excessive difference in the generated image due to the change of the latest frame text (i.e., the target frame text), improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text; at the same time, the constraints of historical text and historical image on the target video generation model to generate the target prediction image can cause the target prediction image to be generated. The generation style or content, etc. are biased towards the historical image, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby improving the overall tone consistency of the video, and an intermediate frame image can be inserted between every two adjacent frame images in the video to be generated, further improving the coherence of the business video indicated by the target text, and further ensuring the quality of the video.

[0137] The embodiment of the present application can generate multiple consistent target prediction images for a main target text during the process of generating a video, so that the contents of the multiple target prediction images are consistent and continuous, thereby ensuring the coherence of the business video composed of the multiple target prediction images, and can provide efficient technical means for the implementation of multiple video scenes, or for the subsequent business video generation given a certain initial image and target text. Potential application directions may include:

[0138] 1. In terms of social media, users can be given a travel picture, a pet picture and a text description (target text) to create a video of subsequent frames, thus providing users with an interesting gameplay experience.

[0139] 2. In terms of virtual display pictures, in e-commerce, merchants can upload photos of a product, and then use the product pictures and different text descriptions (such as the occasions and backgrounds in which the product appears) to freely create product displays, resulting in a variety of visual effect design results for users to choose from.

[0140] For further information, see Figure 3 , Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S301-S305:

[0141] Step S301, obtain the target text, split the target text, and obtain P consecutive frame texts with the video frame sorting order; P is a positive integer.

[0142] In an embodiment of the present application, a computer device can obtain a target text, wherein the target text can come from a video generation request uploaded by a user corresponding to the computer device, or from a video generation request sent by other computer devices, and the target text is used to characterize the video content description of the video to be generated for the user's needs. Further, the computer device can parse the target text, obtain semantic information, and expand the target text according to the semantic information, and split the expanded target text into P continuous frame texts with a video frame sorting order; P is a positive integer. Among them, the P continuous frame texts are respectively used to generate P frame images in the video to be generated, and are respectively used to characterize the description text of the P frame images in the video to be generated. For example, two adjacent continuous frame texts in the P continuous frame texts can be respectively expressed as "the cat begins to lick its front paw with its tongue" and "the cat changes to another paw and continues to lick with its tongue".

[0143] Step S302, obtaining P noise images, obtaining the target frame text traversed in order from the P consecutive frame texts, and determining the target noise image associated with the target frame text from the P noise images.

[0144] In an embodiment of the present application, a computer device can obtain P noise images from a memory, wherein the P noise images can be determined by the computer device based on a noise data pool prepared in advance and noise data obtained from a noise data pool. And each of the P noise images has the same low-frequency signal, and the high-frequency signals between each noise image are different. The computer device obtains the target frame text traversed in order from the P continuous frame texts, that is, the computer device performs image generation processing on the continuous frame texts one by one in the order of the P continuous frame texts. After determining the target frame text, the computer device can assign a corresponding target noise image to the target frame text based on the P noise images. For example, when the target frame text is the first continuous frame text in the P continuous frame texts, the first noise image in the P noise images is determined as the target noise image associated with the target frame text; when the target frame text is the second continuous frame text in the P continuous frame texts, the second noise image in the P noise images is determined as the target noise image associated with the target frame text, and so on, the target noise image associated with the target frame text is determined from the P noise images in order.

[0145] Step S303, input the target noise image into the target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.

[0146] In an embodiment of the present application, a computer device may input a target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model, and obtain a mean vector and a variance vector of the target noise image. Among them, the mean vector may be the average value of all pixel values ​​of the target noise image on each channel, that is, the first-order statistic of the target noise image, and the variance vector may be the average value of the variance of the pixel values ​​of each channel. That is, the second-order statistic of the target noise image. The computer device may randomly sample the mean vector and the variance vector of the target noise image to obtain a potential mean vector and a potential variance vector, and generate a target noise feature based on the potential mean vector and the potential variance vector. Further, in the forward diffusion network layer of the target video generation model, the computer device may obtain a latent variable distribution, wherein the latent variable distribution is a conceptual distribution of added noise, for example, it may be a Gaussian distribution. The computer device may continuously add a random noise vector to the target noise feature in T time steps to obtain a target forward noise vector. Among them, the time step refers to the amplitude of adding noise to the target noise feature based on the latent variable distribution. Noise can refer to unnecessary or redundant interference information present in the image data. When T is large enough, the target forward noise vector can be used to represent an image that is completely noise-containing.

[0147] Step S304, input the business text, business image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image to obtain graphic encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image among P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0148] In the embodiment of the present application, the target video generation model also includes a text encoding layer and a graphic encoding layer. The computer device can perform text segmentation on the target frame text through the text encoding layer to obtain a word unit sequence corresponding to the target frame text. Each word unit in the word unit sequence corresponding to the target frame text is encoded separately to obtain text encoding features. Among them, a word unit in the word unit sequence refers to the smallest basic unit obtained after the target frame text is segmented. The method of text segmentation can be based on word segmentation (word-based), character segmentation (character-based), and subword segmentation (subword-based), which is not limited in the embodiment of the present application. Further, the computer device can perform multimodal feature extraction on the business text and business image through the graphic encoding layer to obtain graphic encoding features. It can be understood that the graphic encoding layer may include a text feature extraction component and an image feature extraction component, wherein the text feature extraction component is used to extract the features of the business text, and the image feature extraction component is used to extract the features of the business image. The process of the computer device extracting the features of the business text and obtaining the text extraction features can be the same as the process of extracting the features of the target frame text through the text encoding layer and obtaining the text encoding features. The computer device fuses the text extraction features and the image extraction features to obtain the image-text coding features. The feature fusion method can be concatenation, element-wise addition, or a more complex interactive fusion technology, which is not limited here. Among them, the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image in the P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text. In other words, the business text is one or more continuous frame texts (i.e., historical texts) sorted before the target frame text in the P continuous frame texts; the business image is the corresponding target prediction image (i.e., historical image) obtained by the computer device based on each business text and performing image generation. It should be noted that when the target frame text is the first continuous frame text in the P continuous frame texts, for the target frame text, there is no previous continuous frame text, that is, there is no business text, and there is no business image. At this time, the business text and business image input by the computer device to the target video generation model can be null value inputs, that is, at this time, the business text is blank null value text, and the business image is a blank null value image.

[0149] Step S305, denoising the target noise image according to the target forward noise vector, text coding features and image-text coding features, obtaining a target predicted image corresponding to the target frame text, and using the target predicted image as the next frame image located in the business image in the video to be generated.

[0150] In an embodiment of the present application, a computer device can generate a fused coding feature based on a text coding feature and a graphic coding feature. For example, when the number of image coding features is 1, and the feature dimensions of the text coding feature and the graphic coding feature are both a, a is a positive integer, and the computer device obtains a spliced ​​coding feature with a feature dimension of 2a after information splicing of the text coding feature and the graphic coding feature. The computer device can perform cross-channel information interaction through a 1×1 convolutional network (Network), reduce the dimension of the spliced ​​coding feature, and obtain a fused coding feature with a feature dimension of a, that is, compress the channel feature of 2a back to a channel feature. Among them, the fused coding feature has the same dimension as the target forward noise vector. Furthermore, the computer device performs reverse diffusion processing on the fused coding feature and the target forward noise vector in the reverse diffusion network layer of the target video generation model to obtain the target prediction noise. The reverse diffusion network layer may include a cross attention subnetwork and a bypass attention subnetwork. The computer device may perform cross attention processing on the fused coding features and the target forward noise vector through the cross attention subnetwork of the reverse diffusion network layer to obtain a first attention processing result; and perform cross attention processing on the fused coding features and the target forward noise vector through the bypass attention subnetwork of the reverse diffusion network layer to obtain a second attention processing result. A first cross attention coefficient and a second cross attention coefficient are obtained, and based on the first cross attention coefficient and the second cross attention coefficient, the first attention processing result and the second attention processing result are weighted to obtain a cross attention weighted result, and the target prediction noise is determined based on the cross attention weighted result.

[0151] Optionally, the number of cross-attention subnetworks and bypass-attention subnetworks may be D, where D is a positive integer. For example, the D cross-attention subnetworks include cross-attention subnetwork 1, cross-attention subnetwork 2, ..., cross-attention subnetwork D, and the D bypass-attention subnetworks include bypass-attention subnetwork 1, bypass-attention subnetwork 2, ..., bypass-attention subnetwork D. In the cross-attention sub-network 1, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, the value vector and the query vector are cross-attention calculated to obtain a first attention processing result 1; in the bypass attention sub-network 1, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, the value vector and the query vector are cross-attention calculated to obtain a second attention processing result 1. The first attention processing result 1 and the second attention processing result 1 are weighted by the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result 1. Further, the computer device can determine the cross-attention weighted result 1 as the query vector in the cross-attention sub-network 2 and the bypass attention sub-network 2, respectively, and use the fused coding feature as the key vector and value vector of the cross-attention function, obtain the first attention processing result 2 based on the cross-attention sub-network 2, obtain the second attention processing result 2 based on the bypass attention sub-network 2, and then obtain the cross-attention weighted result 2, and so on, until the processing of D cross-attention sub-networks and D bypass attention sub-networks is completed, and the cross-attention weighted result D is obtained, and the target prediction noise is determined based on the cross-attention weighted result D. It should be noted that the key vectors and value vectors in the D cross-attention sub-networks and the D bypass attention sub-networks are all fused coding features.

[0152] The computer device can determine the target latent vector based on the target prediction noise and the target forward noise vector, that is, the computer device denoises the target forward noise vector through the target prediction noise to obtain the target latent vector. The computer device can reconstruct the target latent vector through the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated. Further, the computer device obtains the next continuous frame text corresponding to the target frame text in P continuous frame texts, determines the next continuous frame text as the new target frame text, and generates a new target prediction image corresponding to the new target frame text based on the target video generation model. When the P continuous frame texts all pass through the target prediction image corresponding to the target video generation model, that is, P frame images are obtained, the computer device can determine the video to be generated containing P frame images as the business video. Among them, the target latent vector is used to characterize the characteristics of the target prediction image, that is, the potential representation of the target prediction image after noise removal in the latent space.

[0153] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business image and the business text play a role in the process of generating the target prediction image, which makes the generated target prediction image and the historical image (i.e., the newly generated frame and the historical frame in the video to be generated) The content (subject object and object scene, etc.) can be kept consistent, avoiding excessive differences in the generated images due to changes in the text of the previous and next frames, and improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text and composed of multiple target prediction images; at the same time, the constraints of historical texts and historical images on the generation of target prediction images by the target video generation model can prompt the generation style or content of the target prediction image to be biased towards the historical image during generation, so as to ensure that the difference between the target prediction image and the historical image is not large, thereby further improving the overall tone consistency of the Wensheng video and ensuring the quality of the Wensheng video.

[0154] For further information, see Figure 4 , Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S401-S407:

[0155] Step S401, obtain the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer.

[0156] In the embodiment of the present application, the specific implementation process of step S401 can be found in Figure 3 The specific description process in step S301 in will not be repeated here.

[0157] Step S402, obtaining a noise data pool, determining P noise images from the noise data pool, obtaining a target frame text traversed in sequence from the P consecutive frame texts, and determining a target noise image associated with the target frame text from the P noise images.

[0158] In an embodiment of the present application, a computer device can obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer. Among them, the noise data can be strip noise data. The computer device randomly obtains M noise data from the noise data pool, splices the M noise data to obtain a first spliced ​​noise map, performs Fourier transform on the first spliced ​​noise map, obtains a first frequency domain corresponding to the first spliced ​​noise map, and determines the frequency signal in the first frequency domain that is less than the low-frequency threshold as a global low-frequency signal; M is a positive integer less than S. Among them, the specific value of the low-frequency threshold can be determined by the video generation task corresponding to the target text. For example, if the generation task corresponding to the target text is a scene video generation task, the low-frequency threshold can be determined to be 5 Hz, and the global low-frequency signal is a frequency signal with a frequency less than 5 Hz; the low-frequency threshold of the character or pet video generation task can be determined to be 10 Hz.

[0159] For details, please refer to Figure 5 , Figure 5 This is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application. Figure 1 ,like Figure 5As shown, the computer device can randomly obtain M noise data from the noise data pool 501. For example, if M is 3, the computer device randomly obtains 3 noise data from the noise data pool 501 (for example, when S is greater than or equal to 7, the 3 noise data randomly obtained can be: noise data 1, noise data 5, and noise data 7), splices the 3 noise data to obtain a first spliced ​​noise map, performs Fourier transform on the first spliced ​​noise map, obtains a first frequency domain corresponding to the first spliced ​​noise map, and determines the frequency signal less than the low-frequency threshold in the first frequency domain as a global low-frequency signal (that is, obtains a frequency signal less than the low-frequency threshold from the first frequency domain and determines it as a global low-frequency signal). Among them, the Fourier transform can be a 3D Fourier transform (3D FT), and a three-dimensional function or signal can be represented by the sum of its component frequencies through the 3D Fourier transform. For an image, it means converting information in the spatial domain to the frequency domain. In the frequency domain, low-frequency signals are usually concentrated in the low-frequency part of the transformation result, that is, the area close to the origin. The computer device can extract these low-frequency components (ie, global low-frequency signals) by setting a low-frequency threshold or using a low-pass filter.

[0160] Furthermore, the computer device may obtain a first sliding step length, obtain P noise data groups from the S noise data through the first sliding step length, and respectively splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M. Figure 5As shown, when M is equal to 3, the first sliding step is equal to 1, and the first noise data group in the P noise data groups is composed of noise data 1, noise data 2, and noise data 3 in the noise data pool, the second noise data group in the P noise data groups can be composed of noise data 2, noise data 3, and noise data 4 in the noise data pool. The computer device can perform Fourier transform on the P second spliced ​​noise images respectively to obtain the second frequency domains corresponding to the P second spliced ​​noise images, and determine the frequency signals greater than or equal to the low-frequency threshold in the P second frequency domains as high-frequency signals (that is, obtain frequency signals greater than or equal to the low-frequency threshold from the P second frequency domains, and determine them as P high-frequency information, including high-frequency signal 1, high-frequency signal 2, high-frequency signal 3, ..., high-frequency signal P). For example, high-frequency signal 1 can be a frequency signal of the high-frequency part (the part greater than or equal to the low-frequency threshold) obtained by the computer device from the first second frequency domain in the P second frequency domains. The computer device can combine the high-frequency signal in each second frequency domain with the global low-frequency signal to obtain P full-frequency signals, and perform inverse Fourier transform on the P full-frequency signals to obtain P noise images (including noise image 1a, noise image 2a, noise image 3a, ..., noise image Pa). For example, the computer device can combine the high-frequency signal 1 with the global low-frequency signal to obtain the full-frequency signal 1, and then perform inverse Fourier transform on the full-frequency signal 1 to obtain the noise image 1a. Among them, the inverse Fourier transform can be a 3D inverse Fourier transform (3D IFFT), which is used to convert a signal or function from the frequency domain back to its original time domain or space domain. In the one-dimensional case, the inverse Fourier transform is the process of converting frequency domain data back to the time domain; in the multi-dimensional case, such as 3D IFFT, the inverse Fourier transform converts the three-dimensional frequency domain data back to the three-dimensional space domain.

[0161] The computer device may obtain the target frame text traversed in order from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images. For example, the P noise images may be associated with the continuous frame texts in the P continuous frame texts in order, the continuous frame text C1 (the first continuous frame text) is associated with the noise image 1a, the continuous frame text C2 is associated with the noise image 2a, ..., the continuous frame text C P(The last continuous frame text) is associated with the noise image Pa. Then, when the target frame text is the first continuous frame text among P continuous frame texts, the target noise image can be the noise image 1a; when the target frame text is the second continuous frame text among P continuous frame texts, the target noise image can be the noise image 2a. The noise image obtained in this way has a consistent low-frequency signal, thereby ensuring that the subsequently generated target prediction image can maintain the layout environment and style consistency. The inconsistency of the high-frequency signal of the noise image allows the target video generation model to generate each frame image (corresponding to the target prediction image) when generating different editing image content according to the control information (target frame text, business image and business text), thereby achieving the strengthening of the initialization style layout control and improving the overall tone consistency of the video by maintaining the low frequency and randomly initializing the high frequency.

[0162] Optionally, the specific implementation process of the computer device obtaining P noise images may also be: the computer device obtains a noise data pool and a second sliding step length, obtains P noise data groups from S noise data through the second sliding step length, and respectively splices the noise data in the P noise data groups to obtain P noise images. Wherein, the noise data pool includes S noise data; S is a positive integer.

[0163] Step S403, input the target noise image into the target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.

[0164] In the embodiment of the present application, the specific implementation process of step S403 can be found in Figure 3 The specific description process of step S303 in is not repeated here. Figure 6 , Figure 6 is a schematic diagram of a target video generation model structure provided in an embodiment of the present application, such as Figure 6 As shown, the computer device inputs the target noise image into the target video generation model, and extracts the features of the target noise image through the target video generation model to obtain the target noise feature Z. The target noise feature Z is further forward diffused for T time steps (time step length) through the forward diffusion network layer FD to obtain the target forward noise vector Z output by the forward diffusion network layer FD. T Among them, forward diffusion can refer to the process of adding noise to the target noise feature Z, and reverse diffusion can refer to the process of adding noise to the target forward noise vector Z. T The time step is used to indicate the amplitude of the added noise in the forward diffusion and the amplitude of the predicted noise in the backward diffusion.

[0165] Step S404, input the business text, business image and target frame text into the target video generation model, and perform text encoding on the target frame text through the target video generation model to obtain text encoding features; the business image belongs to the frame image that has been generated in the video to be generated; the business text is the continuous frame text used to characterize the business image among P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text.

[0166] In the embodiment of the present application, the specific implementation process of step S404 can be found in Figure 3 The specific description process of obtaining the text encoding feature in step S304 in the text encoding feature is not repeated here. Figure 6 ,like Figure 6 As shown, the computer device inputs the business text, business image and target frame text into the target video generation model, and performs text encoding on the target frame text through the text encoding layer τ to obtain text encoding features. Among them, the business image belongs to the frame image that has been generated in the video to be generated, and the business image can be the last frame image in the video to be generated; the business text is the continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the previous continuous frame text of the target frame text.

[0167] Step S405, extract features from the business image through the image encoding component in the target video generation model to obtain image extraction features; extract features from the business text through the text encoding component in the target video generation model to obtain text extraction features; map the image extraction features and the text extraction features to the same feature space, align the mapped image extraction features and the mapped text extraction features to obtain image-text encoding features.

[0168] In an embodiment of the present application, the target video model includes a multimodal feature extraction network, and the multimodal feature extraction network includes an image encoding component, a text encoding component, and a feature adaptation component. The computer device can extract features from the business image through the image encoding component in the target video generation model to obtain image extraction features. Specifically, the image encoding component can be a vision transformer (ViT) in the BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) model, and the BLIP model is a bootstrap multimodal pre-training model for unified vision-language understanding and generation. The computer device divides the business image into B image blocks through ViT, linearly projects the B image blocks respectively, obtains B image projection features, adds corresponding image position features to the B image blocks in the B image projection features, obtains B image update features, and performs feature fusion on the B image update features to obtain image extraction features corresponding to the business image. Among them, the image position feature is used to characterize the position information of the image block in the business image, that is, the spatial feature. The image position feature can be a fixed-length vector, in which each element identifies whether the image block is located at a specific position. For example, the business image is divided into a 2x2 block grid, then the length of the position encoding vector corresponding to the image position feature will be 4, and the four elements indicate whether the image block is located in the corresponding rows and columns in the image. Among them, the image position feature of the image block in the upper left corner of the business image is [1, 0, 1, 0], indicating that it is in the first row and first column in the business image; the image position feature of the image block in the upper right corner is [1, 0, 0, 1], indicating that it is in the first row and second column in the business image; the image position feature of the image block in the lower left corner is [0, 1, 1, 0], indicating that it is in the second row and first column in the business image; the image position feature of the image block in the lower right corner is [0, 1, 0, 1], indicating that it is in the second row and second column in the business image.

[0169] The text encoding component can be an encoder-decoder (Bidirectional Encoder Representations from Transformers, BERT) based on a self-attention model in the BLIP model. The computer device can use BERT to split the business text to obtain a word-gram sequence corresponding to the business text (including G words, G is a positive integer), and perform vector conversion on each word-gram in the word-gram sequence corresponding to the business text to obtain word embedding vectors corresponding to the G words, and add text position information to each word embedding vector to obtain G word update vectors. Among them, the text position information refers to the information about the position of a word-gram in the business text. The G word update vectors are feature fused to obtain text extraction features. Among them, the business image belongs to the frame image generated in the video to be generated; the business text is a continuous frame text used to characterize the business image in P continuous frame texts, and the business text is a continuous frame text sorted before the target frame text.

[0170] The computer device can map the image extraction features and the text extraction features to the same feature space through a feature adaptation component, and align the mapped image extraction features and the mapped text extraction features to obtain image-text coding features. The image-text coding features and the text coding features corresponding to the target frame text have the same feature dimension. The feature alignment method can be through normalization, scaling, or using a specific loss function (e.g., triplet loss or contrast loss) to ensure that the associated image extraction features and text extraction features are close to each other in the feature space.

[0171] For details, please refer to Figure 6 ,like Figure 6 As shown, the target video generation model includes a multimodal feature extraction network, which includes a multimodal feature extraction component MM (including an image encoding component, a text encoding component) and a feature adaptation component (such as Figure 6 The computer device can extract features from the business image through the image encoding component in the multimodal feature extraction component MM to obtain image extraction features; extract features from the business text through the text encoding component to obtain text extraction features, and further map the image extraction features and the text extraction features to the same feature space through the MLP, align the mapped image extraction features and the mapped text extraction features, and obtain image-text encoding features. Among them, the MLP can be composed of 3 stacked linear plus activation layers (single-layer perceptron). The specific layer composition can be seen in Table 1:

[0172] Table 1

[0173] Layer Name Layer Type Number of layers Linear1+act Linear Layer 1 Dropout1 Activation layer (0.2) 1 Linear2+act Linear Layer 1 Dropout2 Activation layer (0.2) 1 Linear3+act Linear Layer 1 Dropout3 Activation layer (0.2) 1

[0174] As shown in Table 1, MLP includes 3 linear layers (Linear1+act, Linear2+act, Linear3+act) and 3 activation layers (Dropout1, Dropout2, Dropout3). The linear layer usually consists of a weight matrix and a bias vector to perform linear transformation. "Act" is used to indicate that an activation function is applied after the linear layer. The activation function can be ReLU, Sigmoid, or Tanh, etc., which is used to introduce nonlinearity so that the model can learn more complex function mappings. The Dropout layer is a regularization technique used to prevent neural networks from overfitting. In this layer, 20% of the neurons (i.e., a ratio of 0.2) are randomly dropped (i.e., set to zero) in each iteration.

[0175] Step S406, generating fused coding features according to the text coding features and the graphic coding features; in the back diffusion network layer in the target video generation model, based on the fused coding features, back diffusion processing is performed on the target forward noise vector in T time steps respectively to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, a target latent vector is determined; image reconstruction is performed on the target latent vector through the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image located in the business image in the video to be generated.

[0176] In an embodiment of the present application, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text. The specific implementation process of the computer device generating a fused coding feature based on the text coding feature and the image-text coding feature can be: the computer device performs feature splicing on the text coding feature and the image-text coding feature to obtain a spliced ​​coding feature, and performs dimension fusion on the spliced ​​coding feature through the feature fusion network in the target video generation model to obtain a fused coding feature. Among them, the dimension of the fused coding feature is the same as that of the target forward noise vector. For example, the feature fusion network can be a 1×1 convolutional network (Network), and the feature dimensions of the text coding feature and the image-text coding feature are both a, where a is a positive integer. After the computer device performs information splicing on the text coding feature and the image-text coding feature, it obtains a spliced ​​coding feature with a feature dimension of 2a. The computer device can perform cross-channel information interaction through the Network, reduce the dimension of the spliced ​​coding feature, and obtain a fused coding feature with a feature dimension of a. Among them, the fused coding feature has the same dimension as the target forward noise vector. Figure 6As shown, the computer device can input the text encoding features output by the text encoding layer τ and the image and text encoding features output by the feature adaptation component MLP into the feature fusion network Network, perform dimension fusion on the concatenated encoding features through the Network, and output the fused encoding features.

[0177] T time steps include time step T i , i is a positive integer less than or equal to T; further, the computer device may be in the time step T of the back diffusion network layer in the target video generation model i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector.

[0178] Specifically, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each of which includes a cross attention sub-network and a bypass attention sub-network. When T is 3, the T noise prediction networks may include noise prediction network 1, noise prediction network 2, and noise prediction network 3. Noise prediction network 1, noise prediction network 2, and noise prediction network 3 each include a cross attention sub-network and a bypass attention sub-network. The computer device performs the following operations at time step T: i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The specific implementation process of the corresponding initial noise vector can be: the computer device at time step T i In the example, the fused encoded features and iterative noise vector are input to the time step T i The corresponding noise prediction network; through the time step T iThe cross-attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain a first attention processing result. Specifically, the computer device can use the initial noise vector as the query vector in the cross-attention function, the fused coding features as the key vector in the cross-attention function, and the fused coding features as the value vector in the cross-attention function. The computer device determines the product of the query vector and the key vector as the first fused feature through the cross-attention function, obtains the number of dimensions corresponding to the fused coding feature, determines the product of the first fused feature and the inverse of the number of dimensions as the second fused feature, converts the second fused feature into the first activation feature based on the activation sub-function in the cross-attention function, and determines the product of the first activation feature and the value vector as the first attention processing result. Among them, a possible cross-attention function can be seen in formula ①:

[0179]

[0180] As shown in formula ①, Attention(Q,K,V) represents the cross attention function, Q is used to represent the query vector query, K is used to represent the key vector key, and V is used to represent the value vector value; T represents transposition, K T represents the transposed matrix of K, d k Used to indicate the number of dimensions corresponding to the fused encoding features; is the first activated feature, The result is processed by the first attention.

[0181] Furthermore, the computer device passes through the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding features and the iterative noise vector to obtain the second attention processing result. The specific implementation process of obtaining the second attention processing result can refer to the process of obtaining the first attention processing result, which will not be repeated here.

[0182] The computer device may obtain the first cross-attention coefficient and the second cross-attention coefficient, and perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result. Further, based on the cross-attention weighted result, determine the time step T i The corresponding initial noise vector. A possible weighted processing process can be seen in formula ②:

[0183] W=αx1+βx2②

[0184] As shown in formula ②, W is used to represent the cross-attention weighted result, α is used to represent the first cross-attention coefficient, x1 is used to represent the first attention processing result, β is used to represent the second cross-attention coefficient, and x2 is used to represent the second attention processing result.

[0185] The computer device can convert the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the above example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise. i+1 is the time step T among T time steps i The next time step.

[0186] It should be noted that the number of cross-attention sub-networks and bypass-attention sub-networks in the noise prediction network corresponding to each time step can be multiple. Please refer to Figure 6 ,like Figure 6 As shown, the target video generation model includes a back diffusion network layer BD, and the computer device generates a target forward noise vector Z through the back diffusion network layer BD. T Perform back diffusion processing (denoising processing) for T time steps to obtain the target prediction noise Z0 output by the back diffusion network layer BD. In the first time step (i.e., time step T1), the noise prediction network 1 includes a first attention network ∈1 and a second attention network ω1. Among them, the first attention network ∈1 can include multiple cross-attention sub-networks, and the second attention network ω1 can include multiple bypass attention sub-networks. For example, the first attention network ∈1 can include D (D can be equal to 16) cross-attention sub-networks (Note: Figure 6 Only four cross-attention sub-networks are drawn, namely, cross-attention sub-network U1, cross-attention sub-network U2, cross-attention sub-network U3, and cross-attention sub-network U4). The second attention network ω1 may include D (D may be equal to 16) bypass attention sub-networks ( Figure 7Only four bypass attention sub-networks are drawn in the figure, namely bypass attention sub-network V1, bypass attention sub-network V2, bypass attention sub-network V3, and bypass attention sub-network V4). Taking four cross-attention sub-networks and four bypass attention sub-networks as examples, the computer device can use the fused coding features as the key vector and value vector of the cross-attention function in the cross-attention sub-network U1, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, the value vector, and the query vector are cross-attention calculated to obtain a first attention processing result 1c; the computer device can use the fused coding features as the key vector and value vector of the cross-attention function in the bypass attention sub-network V1, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, the value vector, and the query vector are cross-attention calculated to obtain a second attention processing result 1d. The first attention processing result 1c and the second attention processing result 1d are weighted by the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result 1e. Furthermore, the computer device can determine the cross-attention weighted result 1e as the query vector in the cross-attention sub-network U2 and the bypass attention sub-network V2 respectively, and use the fused coding features as the key vector and value vector of the cross-attention function, and obtain the first attention processing result 2c based on the cross-attention sub-network U2, and obtain the second attention processing result 2d based on the bypass attention sub-network V2, and then obtain the cross-attention weighted result 2e, and so on, until the processing of D (for example, 4) cross-attention sub-networks and D bypass attention sub-networks is completed, and the cross-attention weighted result De is obtained. It should be noted that the key vectors and value vectors in the D cross-attention sub-networks and the D bypass attention sub-networks are all fused coding features. The computer device determines the initial noise vector Z corresponding to the first time step based on the cross-attention weighted result De. T-1 . Further, in the remaining T-1 time steps, continue through the first attention network ∈ i and the second attention network ω i The initial noise vector Z corresponding to the first time step T-1 Perform back diffusion processing until the target prediction noise Z0 is obtained. It is understandable that the computer device can reuse the existing SD model, and change the input and add the bypass attention sub-network on the basis of the SD model to obtain the target video generation model, so that when training the model, only the model parameters in the bypass attention sub-network need to be adjusted to obtain the target video generation model, and there is no need to adjust the model parameters in the cross attention sub-network in the SD model, thereby saving resources.

[0187] Among them, the noise prediction network includes several downsampling network layers, one intermediate sampling network layer and several upsampling network layers, such as Figure 6 The cross attention sub-network U1, cross attention sub-network U2, bypass attention sub-network V1, bypass attention sub-network V2 shown in the figure belong to the downsampling network layer, and the cross attention sub-network U3, cross attention sub-network U4, bypass attention sub-network V3, bypass attention sub-network V4 shown in the figure belong to the upsampling network layer. Please also refer to Figure 7 , Figure 7 is a schematic diagram of a downsampling network layer structure provided in an embodiment of the present application, such as Figure 7 As shown, the downsampling network layer includes 2 residual processing layers (ResnetBlock2D), 2 transform processing layers (Transformer2DModel) and 1 downsampling processing unit (DownSample2D). The computer device converts the target forward noise vector Z T, fused coding features and time embedding vector (used to characterize the number of time steps, here taking the first time step as an example), input to the first residual processing layer in the downsampling network layer, obtain the output result of the first residual processing layer, further input the output result to the first transform processing layer, obtain the output result of the first transform processing layer, fused coding features and time embedding vector, input it to the second residual processing layer, obtain the output result and fused coding features of the second residual processing layer, and input it to the second transform processing layer, and input the input result of the second transform processing layer to the downsampling processing unit. The output result of the downsampling processing unit (that is, the output result of the downsampling processing network) is used to input to the next downsampling processing network for subsequent data processing. Among them, the main function of the residual processing layer is to help the downsampling network layer solve the gradient disappearance problem, especially in the deep network. By adding residual connections, the downsampling network layer can more easily learn the identity mapping, that is, directly pass the input to the output without losing information in the deep network. A typical ResnetBlock2D contains two convolutional layers, each of which is usually followed by batch normalization and ReLU activation functions. After the input of ResnetBlock2D passes through these layers, its output is added to the input to form a residual connection. In the image segmentation task, Transformer2DModel allows the target video generation model to consider global context information when processing images through the self-attention mechanism. In the noise prediction network (U-net), Transformer2DModel can be used as part of the encoder or decoder, or as a bridge between the two to provide richer feature representation and context information. DownSample2D is used to reduce the spatial dimension of the input data, that is, the height and width of the image, while possibly increasing the number of channels.

[0188] Please also refer to Figure 8 , Figure 8 is a schematic diagram of a transformation processing layer structure provided in an embodiment of the present application, such as Figure 8 As shown in Figure 1, the change processing layer includes 1 normalization processing layer (GroupNorm), 2 convolution processing layers (Conv1×1), 2 exchange processing layers and 1 transformation processing unit (BasicTransformerBlock). The input of the transformation processing layer can be as follows: Figure 7The output of the residual processing layer shown. The input of the transform processing unit is the fused encoded features and the output of the first exchange processing layer. The normalization processing layer (GroupNorm) is used to divide the channels corresponding to each input feature map (noise image) into multiple groups, and then normalize them within each group. Conv1×1 convolution is usually used to reduce or increase the number of channels of the feature map without changing the spatial dimensions (height and width) of the feature map. The exchange processing layer includes a permute function and a reshape operation, where the permute function is used to rearrange the dimensions of the tensor (feature or vector). For example, a tensor of shape (N, C, H, W), where N is the batch size, C is the number of channels, H is the height, and W is the width, can be rearranged to (N, H, W, C) using permute. This rearrangement can convert image data from a two-dimensional spatial feature map into a serialized form so that it can be input into the transform processing unit, which is useful. The reshape operation is used to change the shape of a tensor without changing its data. In the transformation processing layer, the reshape operation is often used to convert the serialized feature map back to its original 2D or 3D shape for further processing or combining with other parts of the U-Net.

[0189] The structure of the transformation processing unit can be found in Fig. 9 , Fig. 9 is a schematic diagram of a transformation processing unit structure provided in an embodiment of the present application, such as Fig. 9 As shown in Figure 1, the transform processing unit includes three linear processing layers (LayerNorm), a self-attention processing layer (SelfAttention), a cross-attention processing layer (CrossAttention), a bypass attention processing layer and a feedforward processing layer. The input of the transform processing unit can be as follows: Figure 8 The output of the first exchange processing layer shown. Among them, the linear processing layer is used to perform a linear transformation on the input to obtain a linear output. The self-attention processing layer is used to capture the relationship between any two elements in the input sequence. The cross-attention processing layer and the bypass attention processing layer are both used to capture the attention relationship between two sequences (the fused coding feature and the output of the second linear processing layer), that is, the attention relationship between the target forward noise vector or the iterative noise vector and the fused coding feature. The computer device can perform weighted processing on the cross-attention processing result obtained by the cross-attention processing layer and the bypass attention processing result obtained by the bypass attention processing layer to obtain an attention weighted result, and input it into the third linear processing layer of the transformation processing unit, and input the output result of the linear processing layer into the feedforward processing layer. The feedforward processing layer can be used to perform feature transformation on the input sequence, introduce a nonlinear activation function (for example, a ReLU function), and perform nonlinear processing.

[0190] Further, the computer device predicts the noise Z0 and the target forward noise vector Z T Determine the target latent vector, reconstruct the target latent vector through the decoder D in the target video generation model, and obtain the target prediction image Y corresponding to the target frame text. Use the target prediction image Y as the next frame image located in the business image in the video to be generated. Furthermore, the computer device obtains the next continuous frame text corresponding to the target frame text in P continuous frame texts, determines the next continuous frame text as the new target frame text, and generates a new target prediction image corresponding to the new target frame text based on the target video generation model. When the P continuous frame texts all pass through the target prediction image corresponding to the target video generation model, that is, P frame images are obtained, the computer device can obtain a video to be generated including P frame images, wherein the P frame images include the target prediction image Y.

[0191] Step S407: interpolate frames between every two adjacent frames of the video to be generated.

[0192] In an embodiment of the present application, after obtaining the video to be generated, the computer device may also perform interpolation processing between every two adjacent frame images of the video to be generated to further increase the continuity of the video. Specifically, when the video to be generated contains frame images corresponding to P consecutive frame texts, the computer device performs image interpolation on every two adjacent frame images to obtain an intermediate frame image between every two adjacent frame images. Among them, the frame images corresponding to the P consecutive frame texts include the target prediction image. Among them, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P. The computer device can be based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result. j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image.

[0193] Specifically, the computer device can use the optical flow algorithm to calculate the frame image Pj-1 To frame image P j The optical flow vector field and frame image P j+1 To frame image P j+2 The optical flow vector field is used to describe the movement of pixels in space. For example, the frame image P j-1 To frame image P j The optical flow vector field is used to describe the frame image P j-1 The pixel points in the frame image P j-1 The corresponding time to frame image P j The corresponding moment, the operation in space. The computer device can use the quadratic optical flow model based on the frame image P j-1 To frame image P j The optical flow vector field and frame image P j+1 To frame image P j+2 The optical flow vector field is calculated from the pixel point of the frame image P j The corresponding time to frame image P j+1 The optical flow vector field of a certain moment in the corresponding moment is calculated, and the position of the pixel corresponding to that moment is predicted (i.e., the pixel position of the intermediate frame). Further, the computer device uses a differentiable optical flow reversal layer to predict the reverse optical flow, that is, from the intermediate frame back to the frame image P j and from the intermediate frame to the frame image P j+1 The optical flow vector field (i.e., the optical flow prediction result) of the frame image P j , frame image P j+1 And the optical flow prediction results, through image reprojection and interpolation technology, generate the frame image P j and frame image P j+1 Optionally, the computer device may also apply an adaptive flow filter to process the inverted optical flow (reverse optical flow) to reduce possible ringing effects and improve the quality of the intermediate frames. Further, the computer device determines the video to be generated, which includes the frame images corresponding to the P consecutive frame texts and the intermediate frame images, as a business video matching the target text.

[0194] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business image and business text play a role in the process of generating the target prediction image, which enables the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) to maintain consistency in content (subject object and object scene, etc.), avoiding excessive differences in the generated images due to changes in the previous and next frame texts, and improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text and composed of multiple target prediction images. At the same time, the high-frequency signal of the frame-by-frame noise image (i.e., the first spliced ​​noise image) generated by the shift of the noise data pool is combined with the global low-frequency signal of the noise data pool as the frame-by-frame noise image for initialization. The consistency of the low-frequency signal can maintain the layout environment and style consistency. The inconsistency of the high-frequency signal allows the target video generation model to generate different editing image content according to the control information (target frame text, business image and business text) for each frame image, thereby further realizing the low-frequency retention and high-frequency random initialization method to strengthen the initialization style layout control and improve the overall tone consistency of the video. In addition, an intermediate frame image is inserted between every two adjacent frame images in the video to be generated, further improving the coherence of the business video indicated by the target text and the consistency of the overall tone of the video, and ensuring the quality of the video.

[0195] For further information, see Fig.10 , Fig.10 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S1001-S1007:

[0196] Step S1001, obtain the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer.

[0197] In the embodiment of the present application, the specific implementation process of step S1001 can be referred to as follows. Figure 4 The specific description content in step S401 is not repeated here.

[0198] Step S1002, obtain a noise data pool, obtain P noise data groups from the S noise data in the noise data pool through a second sliding step, splice the noise data in the P noise data groups respectively to obtain P noise images, obtain the target frame text traversed in sequence from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images.

[0199] In an embodiment of the present application, a computer device may obtain a noise data pool, wherein the noise data pool includes S noise data; S is a positive integer. The computer device may obtain a second sliding step length, and obtain P noise data groups from the S noise data through the second sliding step length, and respectively splice the noise data in the P noise data groups to obtain P noise images. The number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step length is less than M.

[0200] Specifically, the specific implementation method of the computer device obtaining P noise images can be found in Fig.11 , Fig.11 This is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application. Figure 2 ,like Fig.11 As shown, the computer device can obtain the noise data pool 1101 (with Figure 5 The noise data pool 1101 includes S noise data; the computer device can obtain a second sliding step length, and obtain P noise data groups from the S noise data through the second sliding step length. The specific implementation process of obtaining the P noise data groups can be the same as Figure 5 The number of noise data included in each noise data group is M, for example Figure 6 Where M=3, the computer device can respectively splice the noise data in the P noise data groups to obtain P noise images, including noise image 1b, noise image 2b, noise image 3b, ..., noise image Pb. Wherein, the second sliding step is less than M, that is, between two adjacent noise data groups in the P noise data groups, at least one noise data included is the same. For example, the second sliding step is 2, M=3, the first noise data group in the P noise data groups can be composed of noise data 1, noise data 2, and noise data 3; the second noise data group in the P noise data groups can be composed of noise data 3, noise data 4, and noise data 5; there is a same noise data 3 in both the first noise data group and the second noise data group.

[0201] The computer device obtains the target frame text traversed in order from the P continuous frame texts, and determines the target noise image associated with the target frame text from the P noise images. For example, the P noise images can be associated with the continuous frame texts in the P continuous frame texts in order, the continuous frame text C1 (the first continuous frame text) is associated with the noise image 1b, the continuous frame text C2 is associated with the noise image 2b, ..., the continuous frame text C P (The last continuous frame text) is associated with the noise image Pb. Then, when the target frame text is the first continuous frame text among the P continuous frame texts, the target noise image can be the noise image 1b; when the target frame text is the second continuous frame text among the P continuous frame texts, the target noise image can be the noise image 2b. In this way, the two adjacent noise images among the P noise images obtained have some of the same noise data, and the rest of the noise data changes randomly, thereby generating an initialization noise map (noise image) that maintains a certain proportion of the same, avoiding excessive differences in the styles of the final generated frames due to initialization differences between continuous frames, and improving the overall tone consistency of the video.

[0202] Optionally, the specific implementation process of the computer device obtaining P noise images can also be found in Figure 4 The specific description of determining P noise images from the noise data pool in step S402 shown in FIG.

[0203] Step S1003, input the target noise image into the target video generation model, extract the features of the target noise image through the target video generation model to obtain the target noise features, perform forward diffusion processing on the target noise features to obtain the target forward noise vector.

[0204] In the embodiment of the present application, the specific implementation process of step S1003 can be referred to as follows. Figure 4 The specific description content in step S403 is not repeated here.

[0205] Step S1004, inputting the business text, business image and target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model, and obtaining text encoding features.

[0206] In the embodiment of the present application, the specific implementation process of step S1004 can be referred to as follows. Figure 4 The specific description content in step S404 is not repeated here.

[0207] Step S1005, extract features of the business image through the image encoding component in the target video generation model to obtain image extraction features; extract features of the business text through the text encoding component in the target video generation model to obtain text extraction features; map the image extraction features and the text extraction features to the same feature space, align the mapped image extraction features and the mapped text extraction features to obtain graphic encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text among P continuous frame texts used to characterize the business image, and the business text is a continuous frame text sorted before the target frame text.

[0208] In the embodiment of the present application, the specific implementation process of step S1005 can be referred to as follows. Figure 4 The specific description content in step S405 is not repeated here.

[0209] Step S1006, the unit features in the F graphic coding sub-features are combined into a sequence to be matched; based on the sequence to be matched and the text coding features, the fused coding features are determined; in the reverse diffusion network layer in the target video generation model, based on the fused coding features, the target forward noise vector is respectively subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, the target latent vector is determined; the target latent vector is image reconstructed by the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image located in the business image in the video to be generated.

[0210] In the embodiment of the present application, the number of business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the video to be generated; the graphic coding feature includes F graphic coding sub-features, and one graphic coding sub-feature is obtained by multimodal feature extraction based on a business text and the represented business image.

[0211] For details, see Fig.12 , Fig.12 This is a schematic diagram of a target video generation model structure provided in an embodiment of the present application. Figure 2 ,like Fig.12As shown, the business text includes business text 1 to business text F, and the business image includes business image 1 to business image F. The computer device can extract features from F business images through the image encoding component in the multimodal feature extraction component MM to obtain F image extraction features; extract features from F business texts through the text encoding component to obtain F text extraction features, and further map the one-to-one corresponding image extraction features and text extraction features to the same feature space through the feature adaptation network MLP, align the mapped image extraction features and the mapped text extraction features, and obtain F image-text encoding sub-features. The computer device can form a sequence to be matched with the unit features in the F image-text encoding sub-features. Specifically, the computer device can perform bias processing on the F image-text encoding sub-features through a bitwise addition structure (Add) to obtain F updated image-text sub-features with bias characteristics. Among them, the bias feature is used to characterize that the value range of the feature after the bias is different, allowing the target video generation model to distinguish whether the feature is a feature generated based on text or a feature generated based on both image and text. The computer device obtains the unit features of each updated image and text sub-feature from the F updated image and text sub-features, and composes them into a sequence to be matched. Further, the computer device can perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature. Among them, the computer device can obtain the feature matching degree between each unit feature and the text encoding feature by the method of Euclidean distance or cosine similarity. The computer device can sort each unit feature based on the feature matching degree of each unit feature to obtain a feature matching sequence. The computer device can determine the first A unit features in the feature matching sequence as the target matching features; A is a positive integer. The computer device performs feature splicing on the text encoding features and the target matching features to obtain matching splicing features, and further performs dimension fusion on the matching splicing features through a feature fusion network (Network) to obtain a fused encoding feature. Among them, Network is used to make the feature dimension of the fused encoding feature the same as the dimension of the target forward noise vector. For example, when the feature length of the text coding feature is 77 and A=3, the 77th bit in the text coding feature is not fully used and a portion of the default value is filled. At this time, the computer device can determine the first three unit features in the feature matching sequence as the target matching feature, and replace the target matching feature with the default value of the 74th to 76th bits in the text coding feature to obtain the fused coding feature. The 77th bit in the text coding feature is the default end bit and is not used here.

[0212] The computer device performs back diffusion processing on the target forward noise vector in the back diffusion network layer in the target video generation model based on the fusion coding feature in T time steps to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, the target latent vector is determined; the target latent vector is image reconstructed through the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image of the business image in the video to be generated. For the specific implementation process, please refer to Figure 4 The relevant description in step S406 shown in is not repeated here.

[0213] Step S1007: interpolate frames between every two adjacent frames of the video to be generated.

[0214] In the embodiment of the present application, the specific implementation process of step S1007 can be referred to as follows. Figure 4 The specific description content in step S407 is not repeated here.

[0215] Understandably, Fig.12 For the specific implementation process of each network layer and various components in the model component 1201 in the target video generation model, please refer to the above Figure 4 The corresponding embodiments are not described in detail here.

[0216] Through the above process, the business image, business text, and target frame text can be used to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of multiple business images and multiple business texts play a role in the process of generating the target prediction image, which makes the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) have the same content (subject object and object scene, etc.), avoiding the excessive difference in the generated image due to the change of the previous and next frame text, and improving the similarity between the target prediction image and the historical image, thereby improving the coherence and coordination of the video to be generated based on the target text and composed of multiple target prediction images. At the same time, the frame-by-frame noise image generated by the shift of the noise data pool is used as the initialization, and the two adjacent noise images have a certain proportion of the same information, which can avoid the excessive difference in the style of the final generated previous and next frames due to the initialization difference between the continuous frames generated based on the noise image, thereby strengthening the initialization style layout control and improving the consistency of the overall tone of the video. In addition, an intermediate frame image is inserted between every two adjacent frame images in the video to be generated, further improving the coherence of the business video indicated by the target text and the consistency of the overall tone of the video, and ensuring the quality of the video.

[0217] See also Fig.13 , Fig.13This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 4 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S1301-S1306:

[0218] Step S1301, obtaining a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer.

[0219] In an embodiment of the present application, a computer device may obtain a sample video, obtain N video frames based on the order of video frame sorting in the sample video, and determine the N video frames as N image samples. Wherein, the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other. The computer device may generate text samples for the N image samples respectively through an image description generation model to obtain N text samples. Wherein, the text sample is used to indicate the specific content in the corresponding image sample. The image description generation model may be a generative adversarial network model (Generative Adversarial Networks, GAN), a BLIP model or a scholar multimodal large model (InternVL), or other models that can be used as image generation text descriptions, which are not limited here. The computer device may obtain N noise samples, combine N image samples, N text samples and N noise samples, and obtain a sample sequence. For example, a computer device obtains 10 video frames (image1 to image10) from a sample video, generates corresponding text samples (text1 to text10) for the 10 video frames respectively, and obtains 10 noise samples (noise sample 1 to noise sample 10). The sample sequence obtained by combining them can be expressed as "(image1, text1, noise sample 1), (image2, text2, noise sample 2), (image3, text3, noise sample 3), ..., (image10, text10, noise sample 10)". The specific implementation process of the computer device obtaining N noise samples can be: the computer device obtains a noise data pool; the noise data pool includes S noise data; S is a positive integer. R noise data are randomly obtained from the noise data pool, the R noise data are spliced ​​to obtain a first sample noise map, the first sample noise map is Fourier transformed to obtain a first sample frequency domain corresponding to the first sample noise map, and the frequency signal in the first sample frequency domain that is less than the sample low-frequency threshold is determined as the sample global low-frequency signal; R is a positive integer less than S. Obtain the first sample sliding step length, obtain N sample noise data groups from S noise data through the first sample sliding step length, splice the noise data in the N sample noise data groups respectively, and obtain N second sample noise images; the number of noise data included in each sample noise data group is R. Perform Fourier transform on the N second sample noise images respectively to obtain the second sample frequency domains corresponding to the N second sample noise images respectively, and determine the frequency signals greater than or equal to the sample low-frequency threshold in the N second sample frequency domains as sample high-frequency signals respectively. Combine the sample high-frequency signals in each second sample frequency domain with the sample global low-frequency signal respectively to obtain N sample full-frequency signals, and perform inverse Fourier transform on the N sample full-frequency signals respectively to obtain N noise samples.The specific implementation process of the computer device obtaining N noise samples can also be found in FIG. Figure 4 In step S402 shown in FIG. , P noise images are obtained, or as shown in FIG. Figure 5 The process of obtaining the scene schematic diagram from the noise image shown will not be repeated here.

[0220] Step S1302: determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples.

[0221] In the embodiment of the present application, the computer device sequentially traverses and obtains the target image sample from the N image samples, and determines the target text sample associated with the target image sample from the N text samples. For example, the first target image sample obtained by the computer device from the N image samples is image1, and the target text sample associated with the target image sample determined from the N text samples is text1.

[0222] Step S1303, extracting features of the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain potential noise features, and performing forward diffusion processing on the potential noise features to obtain a sample forward noise vector.

[0223] In the embodiment of the present application, the computer device can determine the target noise sample corresponding to the target image sample from the N noise samples. For example, if the target image sample is image1, the target noise sample can be the first noise sample among the N noise samples. Fig.14 , Fig.14 This is a schematic diagram of an initial video generation model structure provided in an embodiment of the present application. Figure 1 ,like Fig.14 As shown, the computer device can input the target noise sample and the target image sample x into the initial video generation model, extract the target image sample features through the encoder E (for example, it can be an image encoder VAE) in the initial video generation model, and obtain the target image sample features. The target noise sample is extracted through the initial video generation model to obtain a target noise sample vector, and the latent noise feature Z' is determined based on the target image sample features and the target noise sample vector. In the T time steps of the initial forward diffusion network layer FD, the computer device continuously adds the noise vector (target noise sample vector) corresponding to the target noise sample to the latent noise feature to obtain the sample forward noise vector Z T '.

[0224] Step S1304, performing text encoding on the target text sample to obtain a first encoding feature, performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; the first text sample is a text sample associated with the first image sample.

[0225] In the embodiment of the present application, the computer device performs text encoding on the target text sample to obtain the specific implementation process of the first encoding feature, which can be seen in FIG. Figure 3 In step S304 shown in FIG, text encoding is performed on the target frame text to obtain the specific implementation process of the text encoding feature. The computer device extracts multimodal features from the first image sample and the first text sample to obtain the specific implementation process of the second encoding feature, which can be seen in FIG. Figure 3 In step S304 shown in , multimodal feature extraction is performed on business text and business image to obtain the specific implementation process of image-text coding features, which will not be repeated here. Among them, the first image sample is an image sample that is sorted before the target image sample among the N image samples, and is the previous image sample of the target image sample among the N image samples, and the first text sample is a text sample associated with the first image sample. It should be noted that when the target image sample is the first image sample among the N image samples, the first text sample and the first image sample input into the initial video generation model are null value inputs, that is, at this time the first text sample is blank null value text, and the first image sample is a blank null value image. Please also refer to Fig.14 ,like Fig.14 As shown, the initial video generation model also includes a text encoding layer τ and an initial multimodal feature extraction network, and the initial multimodal feature extraction network includes a multimodal feature extraction component MM (including an image encoding component and a text encoding component) and an initial feature adaptation component MLP. The computer device can perform text encoding on the target text sample through the text encoding layer τ to obtain a first encoding feature. The image encoding component in the multimodal feature extraction component MM performs feature extraction on the first image sample to obtain a first image extraction feature; the text encoding component performs feature extraction on the first text sample to obtain a first text extraction feature, maps the first image extraction feature and the first text extraction feature to the same feature space, and aligns the mapped first image extraction feature and the mapped first text extraction feature to obtain a picture-text extraction feature. Further, the computer device performs feature adaptation on the picture-text extraction feature through the initial feature adaptation component MLP to obtain a second encoding feature. Among them, the feature dimension of the second encoding feature is the same as the feature dimension of the first encoding feature.

[0226] Step S1305, performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector.

[0227] In the embodiment of the present application, the computer device can perform feature fusion on the first coding feature and the second coding feature to obtain the target coding feature. The specific implementation process can be referred to as Figure 4 In step S406 shown in FIG, the specific implementation process of generating the fusion coding feature according to the text coding feature and the image-text coding feature is not described here. Fig.14 ,like Fig.14 As shown, the initial video generation model also includes an initial back diffusion network layer BD, the initial back diffusion network layer BD includes a first attention network ∈1 and a second initial attention network ω1, the first attention network ∈1 includes multiple cross attention sub-networks, and the second initial attention network ω1 includes multiple initial bypass attention sub-networks. For example, in the first time step (i.e., time step T1), the first attention network ∈1 may include D (D may be equal to 16) cross attention sub-networks (Note: Fig.14 Only four cross-attention sub-networks are drawn, namely, cross-attention sub-network U1, cross-attention sub-network U2, cross-attention sub-network U3, and cross-attention sub-network U4). The second initial attention network ω1 may include D (D may be equal to 16) initial bypass attention sub-networks ( Fig.14 Only four initial bypass attention sub-networks are drawn, namely initial bypass attention sub-network O1, initial bypass attention sub-network O2, initial bypass attention sub-network O3, and initial bypass attention sub-network O4).

[0228] The computer device can use the target encoding feature as the key vector and value vector of the cross attention function in the cross attention sub-network U1, and the sample forward noise vector Z T 'As the query vector of the cross attention function, the key vector, the value vector and the query vector are cross-attention calculated through the cross attention function to obtain the first sample attention result 1f; the computer device can use the target encoding feature as the key vector and value vector of the cross attention function in the initial bypass attention sub-network O1, and use the sample forward noise vector Z T'As the query vector of the cross-attention function, the cross-attention function is used to perform cross-attention calculation on the key vector, the value vector and the query vector to obtain the second sample attention result 1q. The computer device can obtain the first cross-attention sample coefficient and the second cross-attention sample coefficient, and weight the first sample attention result 1f and the second sample attention result 1q through the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain the sample attention weighted result 1r. Further, the computer device can determine the sample attention weighted result 1r as the query vector in the cross-attention sub-network U2 and the initial bypass attention sub-network O2, respectively, and use the target encoding feature as the key vector and value vector of the cross-attention function, obtain the first sample attention result 2f based on the cross-attention sub-network U2, and obtain the second sample attention result 2q based on the initial bypass attention sub-network O2, and then obtain the sample attention weighted result 2r through the first cross-attention sample coefficient and the second cross-attention sample coefficient, and so on, until the processing of D (for example, 16) cross-attention sub-networks and D initial bypass attention sub-networks is completed, and the sample attention weighted result Dr is obtained. It should be noted that the key vectors and value vectors in the D cross-attention sub-networks and the D initial bypass attention sub-networks are all target encoding features. The computer device determines the initial sample noise vector Z corresponding to the first time step based on the sample attention weighted result Dr T-1 '. Further, in the remaining T-1 time steps, the computer device continues to use the first attention network ∈ i and the second initial attention network ω i For the initial sample noise vector Z corresponding to the first time step T-1 'Perform reverse diffusion processing until the predicted noise sample vector Z0' is obtained.

[0229] Step S1306, adjusting the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0230] In an embodiment of the present application, the initial video generation model also includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature; the initial feature fusion network is used to perform feature fusion on the first coding feature and the second coding feature to obtain a target coding feature. Among them, the initial multimodal feature extraction network includes a multimodal feature extraction component MM (including an image coding component and a text coding component) and an initial feature adaptation component MLP. The initial feature adaptation component MLP is used to perform feature adaptation between the image and text extraction features and the first coding features, so that the adapted image and text extraction features (i.e., the second coding features) have the same feature dimension as the first coding features.

[0231] After obtaining the predicted noise sample vectors corresponding to the N noise samples, the computer device can generate a model loss value according to the N noise samples and the N predicted noise sample vectors. The generation process of the model loss value can be referred to formula ③:

[0232]

[0233] As shown in formula ③, L LDM It is used to represent the model loss value, E represents the expected value, ε(x) is used to represent the encoding result of the encoder E in the initial video generation model encoding the target image sample x, y is used to represent the initial video generation model, based on the target image sample x, the target sample prediction image y is obtained, ∈~Ν(0,1), and is used to represent that the random noise ∈ obeys the standard normal distribution, that is, the Gaussian distribution with a mean of 0 and a variance of 1. In the embodiment of the present application, the random noise ∈ is the target noise sample. t is used to represent the time step, ∈ θ The predicted noise sample vector for the target noise sample output by the initial video generation model, z t Used to represent the sample forward noise vector (i.e. Fig.11 Z T '), c is used to represent the target encoding feature. represents the noise ∈ predicted by the initial video generation model θ The L2 loss between (predicted noise sample vector) and the actual sampled noise ∈ (target noise sample). This loss measures the initial video generation model given the sample forward noise vector z t , time step t, and target encoding feature c, the accuracy of noise prediction. Among them, L2 loss is also called Euclidean loss or mean squared error (MSE), which is used to measure the sum of squares of the difference between the model prediction value and the actual observation value (true value), that is, the sum of squares of the difference between the target noise sample (actual observation value) and the predicted noise sample vector (model prediction value) of the initial video generation model.

[0234] Furthermore, the computer device can adjust the parameters of the initial multimodal feature extraction network, the initial feature fusion network, and the initial bypass attention sub-network based on the model loss value to obtain a target video generation model. The target video model is used to generate a target video indicated by the target text based on the target text. Optionally, in the process of training the initial video generation model, the computer device can use multiple batches of noise samples, image samples, and text samples, set a certain learning rate (for example, 1e-4), and perform multiple rounds of training on the initial video generation model. When each round of training is completed, the model loss value corresponding to the round of training is used to adjust the parameters of the initial video generation model until multiple rounds of training are completed to obtain the final target video generation model.

[0235] Through the above process, the noise data pool is used to obtain N noise samples that are consistent with the low-frequency signal and randomly generated by the high-frequency signal, and the sample forward noise vector is further determined by the target noise sample, so that the initial video generation model can learn how to obtain the predicted noise sample vector based on the sample forward noise vector and the target coding feature, so as to generate an image similar to the target image sample. Among them, the target coding feature is jointly determined by the target text sample, the first image sample (historical sample image) and the first text sample (historical sample text) as joint control information, so that the initial video generation model can learn the association relationship from the historical image to the current frame image (target image sample), so that the trained target video generation model has the ability to generate the content of the previous and next frames in a consistent manner, and ensures that the features corresponding to the business image and the business text can play a role when the target video generation model predicts the target predicted image. Make the target predicted image consistent and coherent with the business image.

[0236] Furthermore, the embodiment of the present application performs task adaptability training on the basis of the existing SD model, reuses the existing SD model, and transforms it on the basis of the existing SD model to obtain an initial video generation model, that is, the input of the SD model is changed on the existing SD model, that is, from being controlled by the target text sample to being jointly controlled by the target text sample, the first text sample and the first image sample; and an initial bypass attention subnetwork is added to the existing SD model, that is, the existing SD model only includes the cross-attention subnetwork, and the initial video generation model includes the cross-attention subnetwork and the initial bypass attention subnetwork, so that the initial video generation model has the ability to generate the content of the previous and next frames consistent, avoiding the mismatch between the target video generation model and the target video generation task caused by non-training methods, and saving training resources in the model training process.

[0237] See also Fig.15 , Fig.15This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 5 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for description. The data processing method may at least include the following steps S1501-S1506:

[0238] Step S1501, obtain a sample video, obtain N video frames based on the video frame sorting order in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same main object and object scene, and the display perspectives of the main objects in the N image samples are different; generate text samples for the N image samples respectively to obtain N text samples; the text samples are used to indicate the specific content in the corresponding image samples; obtain N noise samples, and combine the N image samples, the N text samples and the N noise samples to obtain a sample sequence.

[0239] In an embodiment of the present application, a computer device may obtain a sample video, obtain N video frames based on the sorting order of video frames in the sample video, and determine the N video frames as N image samples. The N image samples all contain the same main object and object scene, and the display perspectives of the main objects in the N image samples are different. The computer device may generate text samples for the N image samples respectively to obtain N text samples. The text samples are used to indicate the specific content in the corresponding image samples. The specific implementation process of the computer device obtaining N noise samples may be: the computer device obtains a noise data pool. The noise data pool includes S noise data; S is a positive integer. Further, the computer device may obtain a second sample sliding step length, obtain N noise data groups from the S noise data through the second sample sliding step length, and splice the noise data in the N noise data groups respectively to obtain N noise samples; wherein the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step length is less than R. Optionally, the specific implementation process of the computer device obtaining N noise samples can also be referred to. Fig.13 The specific acquisition process of the N noise samples in step S1301 in FIG. 13. The computer device may combine the N image samples, the N text samples and the N noise samples to obtain a sample sequence.

[0240] Step S1502: determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples.

[0241] In the embodiment of the present application, the specific implementation process of step S1502 can be referred to as follows. Fig.13 The specific description process in step S1302 shown in will not be repeated here.

[0242] Step S1503, extracting features of the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain potential noise features, and performing forward diffusion processing on the potential noise features to obtain a sample forward noise vector.

[0243] In the embodiment of the present application, the specific implementation process of step S1503 can be referred to as follows. Fig.13 The specific description process in step S1303 shown in is not repeated here.

[0244] Step S1504, performing text encoding on the target text sample to obtain a first encoding feature, performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; the first text sample is a text sample associated with the first image sample.

[0245] In the embodiments of this application, please refer to Fig.16 , Fig.16 This is a schematic diagram of an initial video generation model structure provided in an embodiment of the present application. Figure 2 ,like Fig.16As shown, the computer device can perform text encoding on the target text sample through the text encoding layer τ to obtain the first encoding feature. Wherein, the number of the first text sample and the first image sample is H, H is a positive integer, and the H first image samples are all image samples ranked before the target image sample in the N image samples, that is, the specific value of H is determined by the ranking order of the target image sample in the N image samples. For example, if the target image sample is the fifth image sample in the N image samples, then the value of H is 4, and the first image sample includes the first four image samples in the N image samples (first image sample 1, first image sample 2, first image sample 3, first image sample 4). The first text sample is a text sample associated with the first four image samples, that is, the number of the first text samples is also 4, including the first text sample 1, the first text sample 2, the first text sample 3, and the first text sample 4. The second encoding feature includes H second encoding sub-features, and a second encoding sub-feature is obtained by performing multimodal feature extraction based on a first text sample and the represented first image sample. The computer device can perform multimodal feature extraction on a first text sample and a first image sample represented based on the multimodal extraction component MM in the multimodal feature extraction network in the initial video generation model to obtain a first image sample feature, and perform feature adaptation on the first image sample feature through the initial feature video component MLP in the multimodal feature extraction network to obtain an updated image sample feature. After obtaining H updated image sample features, the computer device biases the H updated image sample features through the bitwise addition structure Add to obtain H second encoding sub-features with bias characteristics. The bias feature is used to characterize that the value range of the feature after the bias is different, allowing the initial video generation model to distinguish whether the feature is generated based on text or based on the image and text.

[0246] Step S1505, performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector.

[0247] In the embodiment of the present application, the computer device can generate a target coding feature according to the first coding feature and the second coding feature. The specific implementation process can be seen in FIG. Fig.10 The specific description process of determining the fusion coding features in step S1006 shown in FIG. 1 is not repeated here.

[0248] Furthermore, the computer device performs reverse diffusion processing on the sample forward noise vector in T time steps based on the target coding feature to obtain the specific implementation process of the predicted noise sample vector. Fig.13 The specific description process in step S1305 shown in will not be repeated here.

[0249] Step S1506, adjusting the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0250] In the embodiment of the present application, the specific implementation process of step S1506 can be referred to as follows. Fig.13 The specific description process in step S1306 shown in will not be repeated here.

[0251] Understandably, Fig.16 For the specific implementation process of each network layer and various components in the initial model component 1601 in the initial video generation model, please refer to the above Fig.13 The corresponding embodiments are not described in detail here.

[0252] Through the above process, N noise samples are obtained by using the noise data pool, and two adjacent noise samples in the N noise samples have a certain proportion of the same information. The sample forward noise vector is further determined by the target noise sample in the N noise samples, so that the initial video generation model can learn how to obtain the predicted noise sample vector according to the sample forward noise vector and the target coding feature, so as to generate an image similar to the target image sample. Among them, the target coding feature is jointly determined by the target text sample, the first image sample (historical sample image) and the first text sample (historical sample text) as joint control information, so that the initial video generation model can learn the association relationship between multiple historical images and the current frame image (target image sample), so that the trained target video generation model has the ability to generate the content of the previous and next frames in a consistent manner, and ensures that the features corresponding to the business image and the business text can play a role when the target video generation model predicts the target predicted image. Make the target predicted image consistent and coherent with the business image.

[0253] For further information, see Fig.17 , Fig.17 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 1 The data processing device 1700 may include: a data acquisition module 11 , a data determination module 12 , a data forward diffusion module 13 , a feature encoding module 14 , an image prediction module 15 , and a video determination module 16 .

[0254] The data acquisition module 11 is used to acquire the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer;

[0255] The data determination module 12 is used to obtain P noise images, obtain the target frame text traversed in order from the P continuous frame texts, and determine the target noise image associated with the target frame text from the P noise images;

[0256] The data forward diffusion module 13 is used to input the target noise image into the target video generation model, extract the features of the target noise image through the target video generation model to obtain the target noise features, and perform forward diffusion processing on the target noise features to obtain the target forward noise vector;

[0257] The feature encoding module 14 is used to input the business text, business image and target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain text encoding features, perform multimodal feature extraction on the business text and business image to obtain image-text encoding features; the business image belongs to the frame image generated in the video to be generated; the business text is the continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the continuous frame text sorted before the target frame text;

[0258] The image prediction module 15 is used to denoise the target noise image according to the target forward noise vector, text encoding features and image-text encoding features, obtain the target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated.

[0259] In a possible implementation, when the data determination module 12 is used to obtain P noise images, the data determination module 12 is specifically used to perform the following operations:

[0260] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0261] Randomly obtain M noise data from the noise data pool, splice the M noise data to obtain a first spliced ​​noise image, perform Fourier transform on the first spliced ​​noise image to obtain a first frequency domain corresponding to the first spliced ​​noise image, and determine a frequency signal less than a low-frequency threshold in the first frequency domain as a global low-frequency signal; M is a positive integer less than S;

[0262] Obtain a first sliding step length, obtain P noise data groups from the S noise data through the first sliding step length, and respectively splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M;

[0263] Performing Fourier transform on the P second spliced ​​noise images respectively to obtain second frequency domains respectively corresponding to the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to a low-frequency threshold as high-frequency signals;

[0264] The high-frequency signals in each second frequency domain are respectively combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are respectively inverse Fourier transformed to obtain P noise images.

[0265] In a possible implementation, when the data determination module 12 is used to obtain P noise images, the data determination module 12 is further used to perform the following operations:

[0266] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0267] A second sliding step is obtained, and P noise data groups are obtained from the S noise data through the second sliding step, and the noise data in the P noise data groups are respectively spliced ​​to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step is less than M.

[0268] In a possible implementation, the feature encoding module 14 is used to extract multimodal features from the business text and the business image. When obtaining the image-text encoding features, the feature encoding module 14 is specifically used to perform the following operations:

[0269] Through the image encoding component in the target video generation model, feature extraction is performed on the business image to obtain image extraction features;

[0270] Through the text encoding component in the target video generation model, feature extraction is performed on the business text to obtain text extraction features;

[0271] The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are aligned to obtain the image-text encoding features.

[0272] In a possible implementation, the image prediction module 15 is used to perform denoising on the target noise image according to the target forward noise vector, the text encoding feature and the image-text encoding feature, and when obtaining the target predicted image corresponding to the target frame text, the image prediction module 15 is specifically used to perform the following operations:

[0273] Generate fused coding features according to text coding features and image-text coding features;

[0274] In the reverse diffusion network layer in the target video generation model, based on the fusion coding features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer;

[0275] Determine a target latent vector based on the target prediction noise and the target forward noise vector;

[0276] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.

[0277] In a possible implementation, the business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; when the image prediction module 15 is used to generate a fusion coding feature according to the text coding feature and the image-text coding feature, the image prediction module 15 is specifically used to perform the following operations:

[0278] Perform feature splicing on the text coding feature and the image-text coding feature to obtain the spliced ​​coding feature;

[0279] The concatenated coding features are dimensionally fused through a feature fusion network in a target video generation model to obtain a fused coding feature; the fused coding feature has the same dimension as the target forward noise vector.

[0280] In a possible implementation, the number of business texts and business images is F, F is a positive integer, and the F business images are all frame images generated in the video to be generated; the image-text coding feature includes F image-text coding sub-features, and one image-text coding sub-feature is obtained by performing multimodal feature extraction based on one business text and the represented business image;

[0281] When the image prediction module 15 is used to generate a fusion coding feature according to the text coding feature and the image-text coding feature, the image prediction module 15 is specifically used to perform the following operations:

[0282] The unit features in the F image-text encoding sub-features are combined into a sequence to be matched;

[0283] Perform feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature, sort each unit feature based on the feature matching degree of each unit feature, and obtain a feature matching sequence;

[0284] The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer;

[0285] The text encoding features and target matching features are concatenated to obtain fused encoding features.

[0286] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module 15 is used to perform reverse diffusion processing on the target forward noise vector in T time steps based on the fusion coding feature to obtain the target prediction noise. The image prediction module 15 is specifically used to perform the following operations:

[0287] At time step T i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;

[0288] Set the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the above example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.

[0289] In a possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network and a bypass attention sub-network; the image prediction module 15 is used to i In the above example, based on the fusion coding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is , the image prediction module 15 is specifically used to perform the following operations:

[0290] At time step T i In the example, the fused encoded features and iterative noise vector are input to the time step T i The corresponding noise prediction network;

[0291] Through the time step T i The cross attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding features and the iterative noise vector to obtain the first attention processing result;

[0292] Through the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross-attention processing on the fused coding features and the iterative noise vector to obtain the second attention processing result;

[0293] Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

[0294] In a possible implementation, the image prediction module 15 is used to perform a first cross attention process on the fused coding feature and the initial noise vector. When the first attention process result is obtained, the image prediction module 15 is specifically used to perform the following operations:

[0295] The initial noise vector is used as the query vector in the criss-cross attention function, the fused encoded features are used as the key vector in the criss-cross attention function, and the fused encoded features are used as the value vector in the criss-cross attention function;

[0296] The product of the query vector and the key vector is determined as the first fused feature through the cross attention function;

[0297] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;

[0298] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the first attention processing result.

[0299] In a possible implementation, the data processing device 1700 further includes a video determination module 16, and the video determination module 16 is specifically configured to perform the following operations:

[0300] When the video to be generated contains frame images corresponding to P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include a target prediction image;

[0301] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.

[0302] In a possible implementation, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; the video determination module 16 is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module 16 is specifically used to perform the following operations:

[0303] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;

[0304] Based on the frame image P j , frame image P j+1 And the optical flow prediction results, generate the frame image P j and frame image P j+1 The intermediate frame images between.

[0305] For further information, see Fig.18 , Fig.18 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 2 The data processing device 1800 may include: a sample acquisition module 21 , a sample determination module 22 , a sample forward diffusion module 23 , a sample encoding module 24 , a sample reverse diffusion module 25 , and a parameter adjustment module 26 .

[0306] The sample acquisition module 21 is used to acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer;

[0307] A sample determination module 22, configured to determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples;

[0308] The sample forward diffusion module 23 is used to extract features of the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain potential noise features, and to perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector;

[0309] The sample encoding module 24 is used to perform text encoding on the target text sample to obtain a first encoding feature, and perform multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; the first text sample is a text sample associated with the first image sample;

[0310] The sample reverse diffusion module 25 is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector;

[0311] The parameter adjustment module 26 is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

[0312] In a possible implementation, when the sample acquisition module 21 is used to acquire a sample sequence, the sample acquisition module 21 is specifically used to perform the following operations:

[0313] Obtain a sample video, obtain N video frames based on the sorting order of video frames in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other;

[0314] Generating text samples for N image samples respectively to obtain N text samples; the text samples are used to indicate specific contents in the corresponding image samples;

[0315] Obtain N noise samples, combine the N image samples, the N text samples, and the N noise samples to obtain a sample sequence.

[0316] In a possible implementation, when the sample acquisition module 21 is used to acquire N noise samples, the sample acquisition module 21 is specifically used to perform the following operations:

[0317] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0318] Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine a frequency signal in the first sample frequency domain that is less than the sample low-frequency threshold as a sample global low-frequency signal; R is a positive integer less than S;

[0319] Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data through the first sample sliding step length, and respectively splice the noise data in the N sample noise data groups to obtain N second sample noise images; the number of noise data included in each sample noise data group is R;

[0320] Performing Fourier transform on the N second sample noise images respectively to obtain second sample frequency domains corresponding to the N second sample noise images respectively, and determining frequency signals in the N second sample frequency domains that are greater than or equal to the sample low-frequency threshold as sample high-frequency signals respectively;

[0321] The sample high-frequency signal in each second sample frequency domain is respectively combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are respectively inverse Fourier transformed to obtain N noise samples.

[0322] In a possible implementation, when the sample acquisition module 21 is used to acquire N noise samples, the sample acquisition module 21 is further used to perform the following operations:

[0323] Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer;

[0324] A second sample sliding step is obtained, and N noise data groups are obtained from the S noise data through the second sample sliding step. The noise data in the N noise data groups are respectively spliced ​​to obtain N noise samples; the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step is less than R.

[0325] In a possible implementation, the initial video generation model includes an initial back diffusion network layer, which includes a cross attention sub-network and an initial bypass attention sub-network; the sample back diffusion module 25 is used to perform back diffusion processing on the sample forward noise vector, the first encoding feature and the second encoding feature, and when the predicted noise sample vector is obtained, the sample back diffusion module 25 is specifically used to perform the following operations:

[0326] generating a target coding feature according to the first coding feature and the second coding feature;

[0327] Through the cross-attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the first sample attention result;

[0328] Through the initial bypass attention sub-network of the initial back-diffusion network layer, the sample forward noise vector and the target encoding feature are cross-attended to obtain the second sample attention result;

[0329] Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

[0330] In a possible implementation, the initial video generation model also includes an initial multimodal feature extraction network and an initial feature fusion network. The initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature; the initial feature fusion network is used to generate a target coding feature based on the first coding feature and the second coding feature; the parameter adjustment module 26 is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples. When the target video generation model is obtained, the parameter adjustment module 26 is specifically used to perform the following operations:

[0331] Generate a model loss value according to N noise samples and predicted noise sample vectors respectively predicted by the N noise samples;

[0332] Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain the target video generation model.

[0333] See also Fig.19 , Fig.19 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.19 As shown, the computer device 1900 in the embodiment of the present application may include: a processor 1901, a network interface 1904 and a memory 1905. In addition, the above-mentioned computer device 1900 may also include: a user interface 1903, and at least one communication bus 1902. Among them, the communication bus 1902 is used to realize the connection and communication between these components. Among them, the user interface 1903 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1903 may also include a standard wired interface and a wireless interface. The network interface 1904 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1905 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1905 may optionally also be at least one storage device located away from the aforementioned processor 1901. As Fig.19 As shown, the memory 1905 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.

[0334] The network interface 1904 may provide a network communication element; the user interface 1903 is mainly used to provide an input interface for the user; and the processor 1901 may be used to call a computer program stored in the memory 1905 to execute the steps in the various embodiments of the present application.

[0335] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program is suitable for being loaded and executed by the processor Figure 3 , Figure 4 , Fig.10 , Fig.13 or Fig.15 For details on the methods provided in each step, please refer to the Figure 3 , Figure 4 , Fig.10 , Fig.13 or Fig.15 The implementation methods provided in each step are not repeated here. In addition, the description of the beneficial effects of adopting the same method is not repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the computer program can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.

[0336] The computer-readable storage medium may be the device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0337] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes Figure 3 , Figure 4 , Fig.10 , Fig.13 or Fig.15 The methods provided in the various optional methods are therefore not described in detail here.

[0338] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0339] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0340] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0341] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer programs. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the computer program executed by the processor of the computer or other programmable device generates a program for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1The computer programs may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the computer program stored in the computer-readable memory produces a product including the program device, or is transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The program device is implemented in the process Figure 1 A process or multiple processes and / or structures Figure 1 These computer programs can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing a computer program executed on the computer or other programmable device for implementing the functions specified in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0342] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0343] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0344] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain a target text, and split the target text to obtain P consecutive frame texts in the video frame sorting order; P is a positive integer; Acquire P noise images, acquire target frame texts traversed in order from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images; Inputting the target noise image into a target video generation model, performing feature extraction on the target noise image through the target video generation model to obtain target noise features, and performing forward diffusion processing on the target noise features to obtain a target forward noise vector; Inputting the business text, the business image and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain text encoding features, performing multimodal feature extraction on the business text and the business image to obtain image-text encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text used to characterize the business image among P continuous frame texts, and the business text is a continuous frame text that is sorted before the target frame text; The target noise image is denoised according to the target forward noise vector, the text encoding feature and the image-text encoding feature to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located at the business image in the video to be generated.

2. The method according to claim 1, characterized in that: The obtaining of P noise images comprises: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Randomly obtain M noise data from the noise data pool, splice the M noise data to obtain a first spliced ​​noise image, perform Fourier transform on the first spliced ​​noise image to obtain a first frequency domain corresponding to the first spliced ​​noise image, and determine a frequency signal less than a low-frequency threshold in the first frequency domain as a global low-frequency signal; M is a positive integer less than S; Obtain a first sliding step length, obtain P noise data groups from the S noise data by using the first sliding step length, and respectively splice the noise data in the P noise data groups to obtain P second spliced ​​noise images; the number of noise data included in each noise data group is M; Performing Fourier transform on the P second spliced ​​noise images respectively to obtain second frequency domains respectively corresponding to the P second spliced ​​noise images, and determining frequency signals in the P second frequency domains that are greater than or equal to the low-frequency threshold as high-frequency signals; The high-frequency signals in each second frequency domain are respectively combined with the global low-frequency signal to obtain P full-frequency signals, and the P full-frequency signals are respectively inverse Fourier transformed to obtain P noise images.

3. The method according to claim 1, characterized in that The obtaining of P noise images comprises: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; A second sliding step is obtained, and P noise data groups are obtained from the S noise data through the second sliding step, and the noise data in the P noise data groups are respectively spliced ​​to obtain P noise images; the number of noise data included in each noise data group is M, M is a positive integer, and the second sliding step is less than M.

4. The method according to claim 1, characterized in that: The performing multimodal feature extraction on the business text and the business image to obtain image-text coding features includes: Extracting features of the business image through the image encoding component in the target video generation model to obtain image extraction features; Extracting features of the business text through a text encoding component in the target video generation model to obtain text extraction features; The image extraction features and the text extraction features are mapped to the same feature space, and the mapped image extraction features and the mapped text extraction features are feature aligned to obtain image-text coding features.

5. The method according to claim 1, characterized in that The denoising process is performed on the target noise image according to the target forward noise vector, the text encoding feature and the image-text encoding feature to obtain a target predicted image corresponding to the target frame text, including: Generate a fused coding feature according to the text coding feature and the image-text coding feature; In the reverse diffusion network layer in the target video generation model, based on the fused coding features, the target forward noise vector is subjected to reverse diffusion processing in T time steps respectively to obtain target prediction noise; T is a positive integer; Determining a target latent vector based on the target prediction noise and the target forward noise vector; The target latent vector is reconstructed by a decoder in the target video generation model to obtain a target predicted image corresponding to the target frame text.

6. The method according to claim 5, characterized in that The business image is the last frame image in the video to be generated, and the business text is the previous continuous frame text of the target frame text; generating a fused coding feature according to the text coding feature and the image-text coding feature includes: Performing feature splicing on the text coding feature and the image-text coding feature to obtain a spliced ​​coding feature; The spliced ​​coding features are dimensionally fused through a feature fusion network in the target video generation model to obtain fused coding features; the fused coding features have the same dimension as the target forward noise vector.

7. The method according to claim 5, characterized in that The number of the business texts and business images is F, where F is a positive integer, and the F business images are all the frame images generated in the video to be generated; the image-text coding feature includes F image-text coding sub-features, and one image-text coding sub-feature is obtained by performing multimodal feature extraction based on one business text and the represented business image; The generating a fused coding feature according to the text coding feature and the image-text coding feature includes: The unit features in the F graphic encoding sub-features are combined into a sequence to be matched; Performing feature matching based on the unit features in the sequence to be matched and the text encoding features to obtain the feature matching degree corresponding to each unit feature, and sorting each unit feature based on the feature matching degree of each unit feature to obtain a feature matching sequence; The first A unit features in the feature matching sequence are determined as target matching features; A is a positive integer; The text encoding feature and the target matching feature are concatenated to obtain a fused encoding feature.

8. The method according to claim 5, characterized in that The T time steps include time step T i , i is a positive integer less than or equal to T; based on the fusion coding feature, the target forward noise vector is respectively subjected to reverse diffusion processing in T time steps to obtain the target prediction noise, including: At the time step T i In the example, based on the fused encoding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step of the T time steps, then the time step T i The corresponding iterative noise vector is the target forward noise vector; The time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector at the time step T i+1 In the example, based on the fused encoding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; the time step T i+1 is the time step T in the T time steps i The next time step.

9. The method according to claim 8, characterized in that The reverse diffusion network layer includes noise prediction networks corresponding to T time steps, each of which includes a cross attention sub-network and a bypass attention sub-network; at the time step T i In the example, based on the fused encoding features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector includes: At the time step T i In the process, the fused coding feature and the iterative noise vector are input into the time step T i The corresponding noise prediction network; By the time step T i The cross attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding feature and the iterative noise vector to obtain a first attention processing result; By the time step T i The bypass attention sub-network in the corresponding noise prediction network performs cross attention processing on the fused coding feature and the iterative noise vector to obtain a second attention processing result; Obtain a first cross-attention coefficient and a second cross-attention coefficient, perform weighted processing on the first attention processing result and the second attention processing result based on the first cross-attention coefficient and the second cross-attention coefficient to obtain a cross-attention weighted result, and determine the time step T based on the cross-attention weighted result. i The corresponding initial noise vector.

10. The method according to claim 9, characterized in that The performing a first cross attention processing on the fused coding feature and the initial noise vector to obtain a first attention processing result includes: Using the initial noise vector as a query vector in a cross-attention function, using the fused encoded features as a key vector in the cross-attention function, and using the fused encoded features as a value vector in the cross-attention function; Determining a product of the query vector and the key vector as a first fused feature through the cross attention function; Obtaining the number of dimensions corresponding to the fused coding feature, and determining the product of the first fused feature and the reciprocal of the number of dimensions as the second fused feature; Based on the activation sub-function in the cross-attention function, the second fused feature is converted into a first activation feature, and the product of the first activation feature and the value vector is determined as a first attention processing result.

11. The method according to claim 1, characterized in that: Also includes: When the video to be generated includes frame images corresponding to the P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include the target prediction image; The video to be generated, which includes the frame images respectively corresponding to the P consecutive frame texts and the intermediate frame images, is determined as a business video matching the target text.

12. The method according to claim 11, characterized in that The frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; performing image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images includes: Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain an optical flow prediction result; the frame image P j+1 is the frame image P in the video to be generated j The next frame image, the frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, the frame image P j-1 is the frame image P in the video to be generated j The previous frame image; Based on the frame image P j , the frame image P j+1 and the optical flow prediction result to generate the frame image P j and frame image P j+1 The intermediate frame images between.

13. A data processing method, characterized in that: The method comprises: Acquire a sample sequence; the sample sequence includes N image samples in a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer; Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples; Extracting features of the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain a potential noise feature, and performing forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; Performing text encoding on the target text sample to obtain a first encoding feature, and performing multimodal feature extraction on the first image sample and the first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample; Performing a reverse diffusion process on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

14. The method according to claim 13, characterized in that The obtaining of the sample sequence comprises: Obtain a sample video, obtain N video frames based on the video frame sorting order in the sample video, and determine the N video frames as N image samples; the N image samples all contain the same main object and object scene, and the display perspectives of the main objects in the N image samples are different from each other; Generating text samples for the N image samples respectively to obtain N text samples; the text samples are used to indicate specific content in the corresponding image samples; Obtain N noise samples, and combine the N image samples, the N text samples, and the N noise samples to obtain a sample sequence.

15. The method according to claim 14, characterized in that The obtaining of N noise samples comprises: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; Randomly obtain R noise data from the noise data pool, splice the R noise data to obtain a first sample noise map, perform Fourier transform on the first sample noise map to obtain a first sample frequency domain corresponding to the first sample noise map, and determine a frequency signal in the first sample frequency domain that is less than a sample low-frequency threshold as a sample global low-frequency signal; R is a positive integer less than S; Obtain a first sample sliding step length, obtain N sample noise data groups from the S noise data by using the first sample sliding step length, and respectively splice the noise data in the N sample noise data groups to obtain N second sample noise images; the number of noise data included in each sample noise data group is R; Performing Fourier transform on the N second sample noise images respectively to obtain second sample frequency domains respectively corresponding to the N second sample noise images, and determining frequency signals in the N second sample frequency domains that are greater than or equal to the sample low-frequency threshold as sample high-frequency signals; The sample high-frequency signal in each second sample frequency domain is respectively combined with the sample global low-frequency signal to obtain N sample full-frequency signals, and the N sample full-frequency signals are respectively inverse Fourier transformed to obtain N noise samples.

16. The method according to claim 14, characterized in that The obtaining of N noise samples comprises: Obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer; A second sample sliding step is obtained, and N noise data groups are obtained from the S noise data through the second sample sliding step. The noise data in the N noise data groups are respectively spliced ​​to obtain N noise samples; the number of noise data included in each noise data group is R, R is a positive integer, and the second sample sliding step is less than R.

17. The method according to claim 13, characterized in that The initial video generation model includes an initial back-diffusion network layer, and the initial back-diffusion network layer includes a cross-attention sub-network and an initial bypass attention sub-network; the back-diffusion processing is performed on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector, including: generating a target coding feature according to the first coding feature and the second coding feature; Performing cross-attention processing on the sample forward noise vector and the target encoding feature through the cross-attention sub-network of the initial back-diffusion network layer to obtain a first sample attention result; Performing cross-attention processing on the sample forward noise vector and the target encoding feature through the initial bypass attention subnetwork of the initial back-diffusion network layer to obtain a second sample attention result; Obtain a first cross-attention sample coefficient and a second cross-attention sample coefficient, perform weighted processing on the first sample attention result and the second sample attention result based on the first cross-attention sample coefficient and the second cross-attention sample coefficient to obtain a sample attention weighted result, and determine a predicted noise sample vector based on the sample attention weighted result.

18. The method according to claim 17, characterized in that The initial video generation model also includes an initial multimodal feature extraction network and an initial feature fusion network, wherein the initial multimodal feature extraction network is used to perform multimodal feature extraction on the first image sample and the first text sample to obtain a second coding feature; and the initial feature fusion network is used to generate a target coding feature according to the first coding feature and the second coding feature; The step of adjusting parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples to obtain a target video generation model comprises: Generate a model loss value according to the N noise samples and predicted noise sample vectors respectively predicted by the N noise samples; Based on the model loss value, the parameters of the initial multimodal feature extraction network, the initial feature fusion network and the initial bypass attention sub-network are adjusted to obtain a target video generation model.

19. A data processing device, characterized in that: The device comprises: The data acquisition module is used to acquire the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer; A data determination module is used to obtain P noise images, obtain a target frame text traversed in order from the P consecutive frame texts, and determine a target noise image associated with the target frame text from the P noise images; A data forward diffusion module is used to input the target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector; A feature encoding module, used for inputting a business text, a business image and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain text encoding features, performing multimodal feature extraction on the business text and the business image to obtain image-text encoding features; the business image belongs to a frame image that has been generated in the video to be generated; the business text is a continuous frame text among P continuous frame texts used to characterize the business image, and the business text is a continuous frame text that is sorted before the target frame text; An image prediction module is used to denoise the target noise image according to the target forward noise vector, the text encoding feature and the image-text encoding feature, obtain a target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located at the business image in the video to be generated.

20. A data processing device, characterized in that: The device comprises: A sample acquisition module, used to acquire a sample sequence; the sample sequence includes N image samples having a video frame sorting order, a text sample associated with each image sample, and N noise samples; N is a positive integer; A sample determination module, used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples; A sample forward diffusion module is used to extract features of a target image sample and a target noise sample corresponding to the target image sample among the N noise samples to obtain a potential noise feature, and to perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; A sample encoding module is used to perform text encoding on a target text sample to obtain a first encoding feature, and to perform multimodal feature extraction on a first image sample and a first text sample to obtain a second encoding feature; the first image sample is an image sample that is sorted before the target image sample among the N image samples; and the first text sample is a text sample associated with the first image sample; A sample reverse diffusion module, used for performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; A parameter adjustment module is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.

Images