Data processing method and device, computer, storage medium and program product
By using feature extraction and denoising processing of noise data pool and video generation model in Wensheng video technology, the background inconsistency caused by text changes in front and back frames is solved, and the video quality and coherence are improved.
Patent Information
- Application Number
- CN202411207473.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-06
AI Technical Summary
When the existing Wensheng video technology generates video frames, the generated foreground and background are inconsistent due to the changes in the text of the front and back frames, resulting in low video quality.
By obtaining the target text, the splitting process obtains continuous frame text in the order of video frame sorting, obtains noise data pool, determines the target noise image associated with the target frame text, and uses the target video generation model for feature extraction and forward diffusion processing, combined with The business text and frame text are denoised to generate corresponding predicted images to ensure background similarity.
It improves the overall background similarity and quality of Wensheng video, ensures the consistency and coordination of the front and back frame images, and avoids excessive generation differences caused by changes in front and back frame text.
Smart Images

Figure CN119090989A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, computer, storage medium and program product. Background Art
[0002] With the rapid development of artificial intelligence technology, especially the continuous progress of generative adversarial networks and diffusion models, the field of image and video generation has ushered in a major breakthrough. Text-to-video technology can generate corresponding images only through text descriptions. The core of this technology is to combine natural language processing with computer vision technology, and use trained models to convert text information into visual content. Based on image generation technology, text-to-video is a product that further expands this capability. Text-to-video requires not only that the model can generate static images, but also that it can generate continuous dynamic images based on text descriptions. At present, with the help of the open source stable diffusion model (SD model), the first frame of the text can be generated under specific control parameters, and then the second frame of the image can be generated under some of the same control parameters until all the images required for the video sequence are generated and the video is generated. However, when the generated text input by the SD model changes with respect to the previous frame of text, it is easy to cause the generated foreground and background to be obviously inconsistent with the previous frame when the next frame is generated, resulting in poor generation effects and low quality of the text-to-video. Summary of the invention
[0003] The embodiments of the present application provide a data processing method, device, computer, storage medium and program product, which can improve the overall background similarity of Vincent video and ensure the quality of Vincent video.
[0004] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:
[0005] Obtain the target text, split the target text, and obtain P consecutive frame texts in the order of video frames; P is a positive integer;
[0006] Obtain the target frame text traversed in order from P consecutive frame texts, obtain the noise data pool, and determine the target noise image associated with the target frame text based on the business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P consecutive frame texts;
[0007] The target noise image is input into the target video generation model, and the target noise image is subjected to feature extraction through the target video generation model to obtain the target noise feature, and the target noise feature is subjected to forward diffusion processing to obtain the target forward noise vector;
[0008] Inputting the business text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the business text to obtain a second text feature;
[0009] According to the target forward noise vector, the first text feature and the second text feature, the target noise image is denoised to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located in the business image in the video to be generated.
[0010] On the one hand, an embodiment of the present application provides another data processing method, the method comprising:
[0011] Obtain N image samples in a video frame sorting order, and a text sample associated with each image sample; N is a positive integer;
[0012] Obtain a noise data pool, and determine N noise samples based on the noise data pool and the N image samples; one noise sample is determined by the noise data pool and one image sample;
[0013] Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples;
[0014] Extract features of the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain potential noise features, and perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector; the target noise sample is determined by the first image sample; the first image sample is the previous image sample of the target image sample in the N image samples;
[0015] Performing text encoding on the target text sample to obtain a first encoding feature, and performing text encoding on the first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample;
[0016] Performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector;
[0017] According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.
[0018] An embodiment of the present application provides a data processing device, the device comprising:
[0019] The data acquisition module is used to acquire the target text, split the target text, and obtain P consecutive frame texts with the video frame sorting order; P is a positive integer;
[0020] A data determination module is used to obtain a target frame text traversed in order from P continuous frame texts, obtain a noise data pool, and determine a target noise image associated with the target frame text based on a business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is a frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts;
[0021] A data forward diffusion module is used to input a target noise image into a target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector;
[0022] A feature encoding module, used for inputting the business text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the business text to obtain a second text feature;
[0023] The image prediction module is used to denoise the target noise image according to the target forward noise vector, the first text feature and the second text feature, obtain the target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located at the business image in the video to be generated.
[0024] In a possible implementation, when the data determination module is used to determine the target noise image associated with the target frame text based on the business image and the noise data pool, the data determination module is specifically used to perform the following operations:
[0025] A target splicing noise map associated with the target frame text is obtained through the noise data pool, and a target noise image associated with the target frame text is generated according to the low-frequency signal corresponding to the business image and the high-frequency signal corresponding to the target splicing noise map.
[0026] In a possible implementation, the noise data pool includes S noise data, where S is a positive integer; the data determination module is used to obtain a target splicing noise image associated with the target frame text through the noise data pool, and when generating a target noise image associated with the target frame text according to a low-frequency signal corresponding to the business image and a high-frequency signal corresponding to the target splicing noise image, the data determination module is specifically used to perform the following operations:
[0027] Obtain a sliding step length, obtain P noise data groups from the S noise data by means of the sliding step length, respectively splice the noise data in the P noise data groups to obtain P second spliced noise images, and determine a target spliced noise image associated with the target frame text from the P second spliced noise images; the number of noise data included in each noise data group is M, and M is a positive integer;
[0028] Performing Fourier transform on the business image to obtain a first frequency domain corresponding to the business image, and determining a frequency signal in the first frequency domain that is less than a low-frequency threshold as a low-frequency signal;
[0029] Performing Fourier transform on the target splicing noise image to obtain a target frequency domain corresponding to the target splicing noise image, and determining a frequency signal in the target frequency domain that is greater than or equal to a low-frequency threshold as a high-frequency signal;
[0030] The high-frequency signal is combined with the low-frequency signal to obtain a full-frequency signal, and the full-frequency signal is inversely Fourier transformed to obtain a target noise image associated with the target frame text.
[0031] In a possible implementation, the image prediction module is used to perform denoising on the target noise image according to the target forward noise vector, the first text feature, and the second text feature. When obtaining the target predicted image corresponding to the target frame text, the image prediction module is specifically used to perform the following operations:
[0032] Perform feature concatenation on the first text feature and the second text feature to obtain a target text feature;
[0033] In the reverse diffusion network layer in the target video generation model, based on the target text features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer;
[0034] Determine a target latent vector based on the target prediction noise and the target forward noise vector;
[0035] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.
[0036] In a possible implementation, the image prediction module is used to perform feature concatenation on the first text feature and the second text feature to obtain the target text feature. The image prediction module is specifically used to perform the following operations:
[0037] Performing bias processing on the second text feature to obtain an updated text feature with a bias characteristic; the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature;
[0038] A target text feature is generated based on the first text feature and the updated text feature.
[0039] In a possible implementation, when the image prediction module is used to generate the target text feature based on the first text feature and the update text feature, the image prediction module is specifically used to perform the following operations:
[0040] Obtaining a context window threshold of a target video generation model, and determining a first text length threshold and a second text length threshold based on the context window threshold; the sum of the first text length threshold and the second text length threshold is equal to the context window threshold;
[0041] Based on the first text length threshold and the first text feature, a first concatenated text feature is generated; the feature length of the first concatenated text feature is equal to the first text length threshold;
[0042] Based on the second text length threshold and the updated text feature, a second concatenated text feature is generated; the feature length of the second concatenated text feature is equal to the second text length threshold;
[0043] The first concatenated text feature and the second concatenated text feature are concatenated to obtain a target text feature.
[0044] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module is used to perform reverse diffusion processing on the target forward noise vector in T time steps based on the target text features to obtain the target prediction noise. The image prediction module is specifically used to perform the following operations:
[0045] At time step T i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;
[0046] Set the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the example, based on the target text features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.
[0047] In a possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network; the image prediction module is used to i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is , the image prediction module is specifically used to perform the following operations:
[0048] At time step T i In the example, the target text features and iterative noise vector are input to the time step T i The corresponding cross-attention sub-network;
[0049] Through the time step T i The corresponding cross-attention sub-network in the noise prediction network performs cross-attention processing on the target text features and the iterative noise vector, and the cross-attention processing results;
[0050] Based on the cross-attention processing results, determine the time step T i The corresponding initial noise vector.
[0051] In a possible implementation, the image prediction module is used to perform cross-attention processing on the target text feature and the iterative noise vector. When the cross-attention processing result is obtained, the image prediction module is specifically used to perform the following operations:
[0052] The initial noise vector is used as the query vector in the cross-attention function, the target text features are used as the key vector in the cross-attention function, and the target text features are used as the value vector in the cross-attention function;
[0053] The product of the query vector and the key vector is determined as the first fused feature through the cross attention function;
[0054] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;
[0055] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the cross attention processing result.
[0056] In a possible implementation, the data processing device further includes a video determination module, and the video determination module is specifically configured to perform the following operations:
[0057] When the video to be generated contains frame images corresponding to P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include a target prediction image;
[0058] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.
[0059] In a possible implementation, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; the video determination module is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module is specifically used to perform the following operations:
[0060] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;
[0061] Based on the frame image P j , frame image P j+1 And the optical flow prediction results, generate the frame image P j and frame image P j+1 The intermediate frame images between.
[0062] On the one hand, an embodiment of the present application provides another data processing device, which includes:
[0063] The image sample acquisition module is used to acquire N image samples in the order of video frames and text samples respectively associated with each image sample; N is a positive integer;
[0064] A noise sample acquisition module is used to acquire a noise data pool, and determine N noise samples based on the noise data pool and N image samples; one noise sample is determined by the noise data pool and one image sample;
[0065] A sample determination module, used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples;
[0066] A sample forward diffusion module is used to extract features from a target image sample and a target noise sample corresponding to the target image sample in N noise samples to obtain a potential noise feature, and to perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; the target noise sample is determined by a first image sample; the first image sample is an image sample previous to the target image sample in the N image samples;
[0067] A sample encoding module, used to perform text encoding on a target text sample to obtain a first encoding feature, and to perform text encoding on a first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample;
[0068] A sample reverse diffusion module is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector;
[0069] The parameter adjustment module is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.
[0070] In a possible implementation, the noise data pool includes S noise data, where S is a positive integer; when the noise sample acquisition module is used to determine N noise samples based on the noise data pool and the N image samples, the noise sample acquisition module is specifically used to perform the following operations:
[0071] Obtain a sample sliding step length, obtain N sample noise data groups from the S noise data through the sample sliding step length, and respectively splice the noise data in the N noise data groups to obtain N first sample noise images; the number of noise data included in each sample noise data group is R, and R is a positive integer;
[0072] Based on the N first sample noise maps and the N image samples, N noise samples are generated.
[0073] In a possible implementation, each image sample corresponds to one of N first sample noise maps, and the N image samples include image samples C i , N first sample noise images include image sample C i The corresponding first sample noise map B i , i is a positive integer less than or equal to N; when the noise sample acquisition module is used to generate N noise samples based on the N first sample noise images and the N image samples, the noise sample acquisition module is specifically used to perform the following operations:
[0074] When the image sample C i When the image sample ranked first among N image samples, the first sample noise map B i , determined as image sample C i The corresponding noise sample;
[0075] When the image sample C i If image sample C is not the first image sample in the N image samples, i-1 Perform Fourier transform to obtain image sample C i-1 The first sample frequency domain corresponding to the first sample frequency domain determines the frequency signal less than the sample low-frequency threshold in the first sample frequency domain as the sample low-frequency signal, and the first sample noise map B i Perform Fourier transform to obtain the first sample noise image B i The corresponding second sample frequency domain, the frequency signal greater than or equal to the sample low-frequency threshold in the second sample frequency domain is determined as the sample high-frequency signal, the sample high-frequency signal is combined with the sample low-frequency signal to obtain a sample full-frequency signal, and the sample full-frequency signal is inverse Fourier transformed to obtain an image sample C i The corresponding noise sample; image sample C i-1 is the image sample C among N image samples i The previous image sample of .
[0076] In a possible implementation, the initial video generation model includes an initial back diffusion network layer; the initial back diffusion network layer is used to perform back diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; the parameter adjustment module is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, and when the target video generation model is obtained, the parameter adjustment module is specifically used to perform the following operations:
[0077] According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain the target video generation model, including:
[0078] Generate a model loss value according to N noise samples and predicted noise sample vectors respectively predicted by the N noise samples;
[0079] The parameters of the initial back diffusion network layer are adjusted based on the model loss value to obtain the target video generation model.
[0080] On the one hand, an embodiment of the present application provides a computer device, including a processor, a memory, and an input and output interface;
[0081] The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that a computer device including the processor executes the method in one aspect of an embodiment of the present application.
[0082] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method in one aspect of the embodiment of the present application.
[0083] In one aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the method provided in various optional ways in one aspect of the embodiment of the present application. In other words, when the computer program is executed by the processor, the method provided in various optional ways in one aspect of the embodiment of the present application is implemented.
[0084] In the embodiment of the present application, the target text is obtained, the target text is split and processed, and P continuous frame texts with video frame sorting order are obtained; P is a positive integer; the target frame text traversed in order is obtained from the P continuous frame texts, the noise data pool is obtained, and the target noise image associated with the target frame text is determined based on the business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts. The target noise image is input into the target video generation model, and the target noise image is feature extracted by the target video generation model to obtain the target noise feature, and the target noise feature is forward diffused to obtain the target forward noise vector; the business text and the target frame text are input into the target video generation model, and the target frame text is text encoded by the target video generation model to obtain the first text feature, and the business text is text encoded to obtain the second text feature; the feature extraction method can be fully utilized to extract the features of the target frame text and the historical text (business text). According to the target forward noise vector, the first text feature and the second text feature, the target noise image is denoised to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located in the business image in the video to be generated. By combining the business text with the next frame text (target frame text) to jointly constrain and guide the target video generation model, it is ensured that the main objects of the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) can be kept consistent; by generating a target noise image based on the business image, part of the information of the business image exists in the target noise image, which ensures that the generated target prediction image and the object scene in the historical image can be kept consistent or highly similar, thereby ensuring the background similarity between the target prediction image and the business image, thereby avoiding excessive differences between the generated previous and next frame images due to changes in the previous and next frame texts, improving the similarity between the target prediction image and the historical image, and thus improving the coherence and coordination of the video to be generated based on the target text; at the same time, the historical image constrains the target noise features of the target noise image, and the historical text further constrains the target video generation model to generate the target prediction image, which can cause the background (object scene), generation style or content of the target prediction image to be biased towards the historical image when it is generated, so as to ensure that the difference between the historical image and the target prediction image as the previous and next frames is not large, thereby improving the overall background similarity of the video (that is, the overall tone remains consistent), and further ensuring the quality of the video. BRIEF DESCRIPTION OF THE DRAWINGS
[0085] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0086] Figure 1 This is a network interaction architecture diagram provided by an embodiment of the present application;
[0087] Figure 2 It is a scenario schematic diagram of a data processing method provided in an embodiment of the present application;
[0088] Figure 3 This is a data processing method provided by the embodiment of the present application. Figure 1 ;
[0089] Figure 4 This is a data processing method provided by the embodiment of the present application. Figure 2 ;
[0090] Figure 5 This is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application;
[0091] Figure 6 It is a schematic diagram of a target video generation model structure provided in an embodiment of the present application;
[0092] Figure 7 It is a schematic diagram of a downsampling network layer structure provided in an embodiment of the present application;
[0093] Figure 8 is a schematic diagram of a transformation processing layer structure provided in an embodiment of the present application;
[0094] Fig. 9 is a schematic diagram of a transformation processing unit structure provided in an embodiment of the present application;
[0095] Fig.10 This is a data processing method provided by the embodiment of the present application. Figure 3 ;
[0096] Fig.11 This is a schematic diagram of a noise sample acquisition scenario provided in an embodiment of the present application;
[0097] Fig.12 is a schematic diagram of the structure of an initial video generation model provided in an embodiment of the present application;
[0098] Fig.13 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 1 ;
[0099] Fig.14 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 2 ;
[0100] Fig.15 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0101] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0102] Among them, if it is necessary to collect object (such as user, etc.) data in this application, a prompt interface or pop-up window will be displayed before or during the collection. The prompt interface or pop-up window is used to prompt the user that certain data is currently being collected. Only after the user confirms the prompt interface or pop-up window, the relevant steps of data acquisition will be started, otherwise the process will end. Moreover, the acquired user data will be used in reasonable and legal scenarios or purposes. Optionally, in some scenarios where user data needs to be used but has not been authorized by the user, authorization can be requested from the user, and the user data can be used when the authorization is passed.
[0103] It is understandable that in the specific implementation of this application, the user data involved, when the following embodiments of this application are applied to specific products or technologies, needs to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of the relevant regions.
[0104] In the examples of this application, see Figure 1 , Figure 1 is a network interaction architecture diagram provided by an embodiment of the present application, such as Figure 1As shown, the network interaction architecture diagram may include a business server 101 and a terminal device cluster, and the terminal device cluster may include terminal device 102a, terminal device 102b, terminal device 102c, ..., terminal device 102n, wherein there may be a communication connection between the terminal device clusters, for example, there is a communication connection between terminal device 102a and terminal device 102b, and there is a communication connection between terminal device 102a and terminal device 102c. At the same time, any terminal device in the terminal device cluster may have a communication connection with the business server 101, for example, there is a communication connection between terminal device 102a and business server 101, wherein the above communication connection does not limit the connection method, and may be directly or indirectly connected through a wired communication method, or directly or indirectly connected through a wireless communication method, or through other methods, and this application does not make any restrictions here.
[0105] It should be understood that Figure 1 Each terminal device in the terminal device cluster shown in the figure can be installed with an application client having a data processing function. When the application client runs in each terminal device, it can be respectively connected to the above Figure 1 The business servers 101 shown in the figure interact with each other so that the business server 101 can receive business data from each terminal device. Among them, the application client can be a social application, instant messaging application, live broadcast application, short video application, video application, music application, shopping application, novel application, browser, etc., which has the function of displaying text, images, audio, video and other data information. Among them, the application client can be an independent client or an embedded sub-client (for example, application applet, browser web page access, etc.) integrated in a client (for example, instant messaging client, social client, video client, etc.), which is not limited here.
[0106] like Figure 1As shown, any terminal device in the terminal device cluster can send a video generation request containing a target text to the business server 101. Taking the terminal device 102a as an example, the terminal device 102a can send the target text to the business server 101 through the video generation request. The target text is the description text of the video to be generated indicated by the video generation request. The business server 101 can receive the video generation request sent by the terminal device 102a, obtain the target text from the video generation request, expand the target text based on the semantic information of the target text, and split the expanded target text to obtain P continuous frame texts with a video frame sorting order; P is a positive integer. The P continuous frame texts are respectively used to generate P frame images in the video to be generated, and are respectively used to characterize the description texts of the P frame images in the video to be generated. It can be understood that in order to maintain the coherence of the video to be generated, the difference between two adjacent continuous frame texts in the P continuous frame texts is small, that is, the text description content or style of the two continuous frame texts is similar. For example, the target text can be represented as "a cat is chasing a butterfly on the grass, then suddenly jumps onto a stone, and finally sits on the stone and licks its paws.", where two adjacent continuous frame texts can be represented as "the cat starts to lick its front paw with its tongue" and "the cat switches to the other paw and continues to lick it with its tongue" respectively.
[0107] The business server 101 can traverse the P continuous frame texts in order to determine the target frame text. For example, the first target frame text can be the first continuous frame text in the P continuous frame texts, and the second target frame text can be the second continuous frame text in the P continuous frame texts. The business server 101 can obtain a noise data pool, which can be composed of S noise data, where S is a positive integer. Further, the business server 101 can determine the target noise image associated with the target frame text based on the business image and the noise data in the noise data pool. Among them, the business image is the last frame image in the video to be generated, that is, the latest frame image (historical image), and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts. That is, when the target frame text is the second continuous frame text in the P continuous frame texts, the business text is the first continuous frame text in the P continuous frame texts. The service server 101 can input the target noise image into the target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain the target noise feature, perform forward diffusion processing on the target noise feature to obtain the target forward noise vector. The service server 101 can input the service text and the target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain the first text feature, and perform text encoding on the service text to obtain the second text feature.
[0108] The business server 101 can perform denoising on the target noise image according to the target forward noise vector, the first text feature and the second text feature, obtain the target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated. When the target prediction image corresponding to the last continuous frame text is generated (that is, when the target prediction images corresponding to P continuous frame texts are obtained), the business server 101 can obtain the business video indicated by the target text, and the business video includes frame images corresponding to P continuous frame texts, that is, P frame images. The business server 101 can send the business video to the terminal device 102a so that the terminal device 102a can display the business video.
[0109] The embodiment of the present application determines the target noise image corresponding to the target frame text through the business image, so that the features of the business image can play a role in the subsequent process of generating the target prediction image; fully utilizes the feature extraction method to extract the features of the target frame text and the historical text (business text), and jointly constrains and guides the target video generation model through the business text and the target frame text to generate a target prediction image for the target frame text, so that the features of the business text can play a role in the process of generating the target prediction image, ensuring that the main objects of the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) can be kept consistent; by generating a target noise image based on the business image, a target prediction image corresponding to the target frame text is generated, ensuring that the generated target prediction image and the historical image object scene can be kept consistent. The similarity between the target predicted image and the historical image is improved, thereby improving the coherence and coordination of the video to be generated based on the target text; at the same time, the historical image constrains the target noise characteristics of the target noise sample, and the historical text further constrains the target video generation model to generate the target predicted image, which can cause the background (object scene), generation style or content of the target predicted image to be biased towards the historical image during generation, so as to ensure that the difference between the historical image and the target predicted image as the previous and next frames is not large, thereby improving the overall background similarity of the Wensheng video (referring to the video to be generated or the business video generated based on the target text) (that is, the overall tone remains consistent), and further ensuring the quality of the Wensheng video.
[0110] It is understandable that the terminal device mentioned in the embodiments of the present application may also be a computer device, and the computer device in the embodiments of the present application includes but is not limited to a terminal device or a server. In other words, the computer device may be a server or a terminal device, or it may be a system composed of a server and a terminal device. Among them, the terminal device mentioned above may be an electronic device, including but not limited to mobile phones, tablet computers, desktop computers, laptop computers, PDAs, vehicle-mounted devices, augmented reality / virtual reality (AR / VR) devices, helmet displays, smart TVs, wearable devices, smart speakers, digital cameras, cameras and other mobile Internet devices (mobile internet device, MID) with network access capabilities, or terminal devices in scenarios such as trains, ships, and flights. As Figure 1 As shown in , the terminal device may be a mobile phone (as shown in terminal device 102a), a desktop computer (as shown in terminal device 102b), a tablet computer (as shown in terminal device 102c) or a laptop computer (as shown in terminal device 102n), etc. Figure 1 Only some of the devices are listed. The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, vehicle-road collaboration, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0111] Optionally, the data involved in the embodiments of the present application can be stored in a computer device, or the data can be stored based on cloud storage technology or a blockchain network, which is not limited here.
[0112] For details, see Figure 2 , Figure 2 Schematic diagram of a data processing method provided in an embodiment of the present application. Figure 2 As shown, the service server can obtain the target text in the video generation request sent by the terminal device, wherein the service server can be Figure 1 The service server 101 shown, the terminal device can be Figure 1Any terminal device in the terminal device cluster shown, here taking terminal device 102a as an example. The user corresponding to terminal device 102a inputs a video description (target text) and initiates a video generation request. The service server 101 can obtain the target text in the video generation request sent by terminal device 102a, expand the target text, and split the expanded target text into P continuous frame texts (including continuous frame text A1, continuous frame text A2, ..., continuous frame text A P ), wherein the P continuous frame texts have a video frame sorting order. The business server 101 can obtain the target frame text traversed in order from the P continuous frame texts, that is, the first target frame text can be the continuous frame text A1. The business server 101 can obtain the noise data pool, and determine the target noise image associated with the target frame text based on the business image and the noise data pool. Among them, the business image is the last frame image in the video to be generated, that is, the latest frame image (historical image), and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts. That is, when the target frame text is the second continuous frame text (continuous frame text A2) among the P continuous frame texts, the business text is the first continuous frame text (continuous frame text A1) among the P continuous frame texts. The noise data pool includes S noise data, S is a positive integer, and the business server 101 can combine the business image and one or more noise data in the noise data pool to generate a target noise image associated with the target frame text.
[0113] The service server 101 can input the target noise image into the target video generation model, extract the target noise image through the target video generation model to obtain the target noise feature, perform forward diffusion processing on the target noise feature, and obtain the target forward noise vector. Among them, the target video generation model can be obtained by modifying the SD model, and the service server 101 can perform forward diffusion processing on the target noise feature through the forward diffusion network layer of the target video generation model to obtain the target forward noise vector. The forward diffusion network layer can be the forward diffusion network layer in the SD model. The service server 101 can input the service text and the target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain the first text feature; perform feature encoding on the service text to obtain the second text feature. It should be noted that when the target frame text is the first continuous frame text among P continuous frame texts (i.e., continuous frame text A1), the business text input into the target video generation model by the business server 101 can be a null value input, that is, the input business text is a blank null value text. At this time, when the business server 101 determines the target noise image based on the business image and the noise data pool, the business image used is a blank null value image, that is, the final target noise image is generated only by the noise data in the noise data pool.
[0114] Furthermore, the business server 101 can generate a model for the target video and, based on the target forward noise vector, the first text feature, and the second text feature, perform denoising on the target noise image to obtain a target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located at the business image in the video to be generated. When the business server 101 obtains target prediction images corresponding to P consecutive frame texts, that is, P target prediction images (including target prediction image F1, target prediction image F2, ..., target prediction image F P ), the final video to be generated can be obtained. The final video to be generated includes P frame images, the target prediction image F1 is the target prediction image corresponding to the continuous frame text A1, the target prediction image F2 is the target prediction image corresponding to the continuous frame text A2, and the target prediction image F P A is a continuous frame text PThe corresponding target predicted image. Optionally, the business server 101 can further perform frame insertion processing on the final video to be generated, further improve the inter-frame coherence of the video to be generated, and obtain a business video. That is, the business server 101 inserts one or more intermediate frame images into each two adjacent frame images of the P frame images of the final video to be generated, so that the transition between the two adjacent frame images is more natural, and the obtained business video is more coherent. Among them, the business video includes the P frame images in the video to be generated, and the intermediate frame images inserted between each two adjacent frame images. The business server 101 can send the business video to the terminal device 102a, so that the terminal device 102a can display the business video to the user at the front end.
[0115] Through the above process, the target noise image corresponding to the target frame text can be determined based on the business image, so that the features of the business image can play a role in the subsequent process of generating the target prediction image; make full use of the feature extraction method to extract the features of the target frame text and the historical text (business text), and use the business text and the target frame text to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business text can play a role in the process of generating the target prediction image, thereby ensuring that the main objects of the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) can be kept consistent; by generating a target noise image based on the business image, the target prediction image corresponding to the target frame text is generated, thereby ensuring that the object scenes of the generated target prediction image and the historical image can be kept consistent. The similarity between the target predicted image and the historical image is improved, thereby improving the coherence and coordination of the video to be generated based on the target text; at the same time, the historical image constrains the target noise characteristics of the target noise sample, and the historical text further constrains the target video generation model to generate the target predicted image, which can cause the background (object scene), generation style or content of the target predicted image to be biased towards the historical image during generation, thereby improving the overall background similarity of the Wensheng video (referring to the video to be generated or the business video generated based on the target text) (that is, the overall tone remains consistent); and an intermediate frame image can be inserted between every two adjacent frame images in the video to be generated, further improving the coherence of the business video indicated by the target text, and further ensuring the quality of the Wensheng video.
[0116] The embodiment of the present application can generate multiple consistent target prediction images for a main target text during the process of generating a video, so that the contents of the multiple target prediction images are consistent and continuous, thereby ensuring the coherence of the business video composed of the multiple target prediction images, and can provide efficient technical means for the implementation of multiple video scenes, or for the subsequent business video generation given a certain initial image and target text. Potential application directions may include:
[0117] 1. In terms of social media, users can be given a travel picture, a pet picture and a text description (target text) to create a video of subsequent frames, thus providing users with an interesting gameplay experience.
[0118] 2. In terms of virtual display pictures, in e-commerce, merchants can upload photos of a product, and then use the product pictures and different text descriptions (such as the occasions and backgrounds in which the product appears) to freely create product displays, resulting in a variety of visual effect design results for users to choose from.
[0119] For further information, see Figure 3 , Figure 3 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 1 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S301-S305:
[0120] Step S301, obtain the target text, split the target text, and obtain P consecutive frame texts with the video frame sorting order; P is a positive integer.
[0121] In an embodiment of the present application, a computer device can obtain a target text, wherein the target text can come from a video generation request uploaded by a user corresponding to the computer device, or from a video generation request sent by other computer devices, and the target text is used to characterize the video content description of the video to be generated in response to user needs. Further, the computer device can parse the target text, obtain semantic information, and expand the target text according to the semantic information, and split the expanded target text into P continuous frame texts (including continuous frame text A1, continuous frame text A2, ..., continuous frame text A P); P is a positive integer. Among them, the P consecutive frame texts are used to generate the P frame images in the video to be generated, and are used to represent the description texts of the P frame images in the video to be generated. For example, two adjacent consecutive frame texts text1 and text2 in the P consecutive frame texts can be expressed as "he is swimming" and "he is swimming forward" respectively. That is, compared with text1, text2 adds the advancement of the action (forward, or keeps the same action, that is, text2 is still he is swimming). This action advancement information is reflected in the similarity of the sentence structure and the same verbs in the previous and next frames of the video, and the previous and next descriptions of the text. The model is trained using graphic samples with such information, so that the target video generation model can be gradually transferred through text - under the control of text1 to text2, the target noise image (with the low-frequency signal of the previous frame) is gradually denoised to obtain the next frame of the action change.
[0122] Step S302, obtain the target frame text traversed in order from P consecutive frame texts, obtain the noise data pool, and determine the target noise image associated with the target frame text based on the business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P consecutive frame texts.
[0123] In an embodiment of the present application, a computer device obtains a target frame text traversed in order from P continuous frame texts, that is, the computer device performs image generation processing on the continuous frame texts one by one in the order of the P continuous frame texts. The computer device can obtain a noise data pool, and determine a target noise image associated with the target frame text based on the business image and the noise data pool. Among them, the business image is the last frame image in the video to be generated, and the business image is a frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts. Specifically, the computer device can obtain a target noise data group associated with the target frame text from the noise data pool, combine the low-frequency signal in the business image and the high-frequency signal in the target noise data group, and obtain a target noise image. Among them, the target noise data group may include one or more noise data. For example, when the second continuous frame text (continuous frame text A2) among P continuous frame texts is used as the target frame text, the computer device generates a target noise image corresponding to the continuous frame text A2 based on the business image 1 and the noise data pool, and generates a frame image a for the continuous frame text A2 based on the target noise image corresponding to the continuous frame text A2; when the third continuous frame text (continuous frame text A3) among P continuous frame texts is used as the target frame text, the computer device generates a target noise image corresponding to the continuous frame text A3 based on the business image 2 (i.e., frame image a) and the noise data pool, and generates a frame image b for the continuous frame text A3 based on the target noise image corresponding to the continuous frame text A3. Among them, the business image 1 is a frame image generated by the computer device for the continuous frame text A1. In the process of the computer device generating a frame image for the continuous frame text A1, the business image used can be a blank null value image, that is, the target noise image used in this process is only determined by the noise data pool.
[0124] It can be understood that when P consecutive frame texts are respectively used as target frame texts, in the corresponding target noise images (i.e., in the P target noise images), the low-frequency signal of each target noise image is determined by the business image corresponding to the corresponding target frame text, so that the low-frequency signal of each target noise image has a certain correlation, and the high-frequency signal of each target noise image is different. For example, when the continuous frame text A2 is used as the target frame text, the business image is the frame image represented by the continuous frame text A1 (which can be represented as business image 1), and when the computer device performs image generation processing on the continuous frame text A2, the low-frequency information in the target noise image 1 used is determined by the business image 1; when the continuous frame text A3 is used as the target frame text, the business image is the frame image represented by the continuous frame text A2 (i.e., the image obtained by the computer device performing image generation processing on the continuous frame text A2 can be represented as business image 2), and when the computer device performs image generation processing on the continuous frame text A3, the low-frequency information in the target noise image 2 used is determined by the business image 2. The business image 2 is determined by combining the low-frequency signal in the business image 1 with the high-frequency signal in the target noise data group, so that the target noise image 1 determined by the business image 1 and the target noise image 2 determined by the business image 2 are correlated, and further the frame image a (business image 2, or the frame image corresponding to the continuous frame text A2) generated by the target noise image 1 and the frame image b (or the frame image corresponding to the continuous frame text A3) generated based on the target noise image 2 have background similarity, that is, the two adjacent frame images in the video to be generated have background similarity, thereby improving the overall tone consistency of the video to be generated.
[0125] Step S303, input the target noise image into the target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.
[0126] In an embodiment of the present application, a computer device may input a target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model, and obtain a mean vector and a variance vector of the target noise image. Among them, the mean vector may be the average value of all pixel values of the target noise image on each channel, that is, the first-order statistic of the target noise image, and the variance vector may be the average value of the variance of the pixel values of each channel. That is, the second-order statistic of the target noise image. The computer device may randomly sample the mean vector and the variance vector of the target noise image to obtain a potential mean vector and a potential variance vector, and generate a target noise feature based on the potential mean vector and the potential variance vector. Further, in the forward diffusion network layer of the target video generation model, the computer device may obtain a latent variable distribution, wherein the latent variable distribution is a conceptual distribution of added noise, for example, it may be a Gaussian distribution. The computer device may continuously add a random noise vector to the target noise feature in T time steps to obtain a target forward noise vector. Among them, the time step refers to the amplitude of adding noise to the target noise feature based on the latent variable distribution. Noise can refer to unnecessary or redundant interference information present in the image data. When T is large enough, the target forward noise vector can be used to represent an image that is completely noise-containing.
[0127] Step S304, inputting the business text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the business text to obtain a second text feature.
[0128] In an embodiment of the present application, the target video generation model also includes a text encoding layer, and the computer device can use the text encoding layer to perform text splitting on the target frame text to obtain a word-unit sequence corresponding to the target frame text. Each word-unit in the word-unit sequence corresponding to the target frame text is encoded separately to obtain a first text feature. Among them, a word-unit in the word-unit sequence refers to the smallest basic unit obtained after the target frame text is split. The method of text splitting can be based on word splitting (word-based), character splitting (character-based), and subword splitting (subword-based), which is not limited in the embodiment of the present application. Further, the computer device uses the text encoding layer to perform text splitting on the business text to obtain a word-unit sequence corresponding to the business text. Each word-unit in the word-unit sequence corresponding to the business text is encoded separately to obtain a second text feature. It should be noted that when the target frame text is the first continuous frame text in P continuous frame texts, for the target frame text, there is no previous continuous frame text, that is, there is no business text. At this time, the business text input by the computer device to the target video generation model can be a null value input, that is, the business text is a blank null value text at this time.
[0129] Step S305, denoising the target noise image according to the target forward noise vector, the first text feature and the second text feature, obtaining a target predicted image corresponding to the target frame text, and using the target predicted image as the next frame image located in the business image in the video to be generated.
[0130] In an embodiment of the present application, a computer device may perform feature splicing on a first text feature and a second text feature to obtain a target text feature. Specifically, the computer device may perform bias processing on the second text feature to obtain an updated text feature with a bias characteristic; the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature. For example, the computer device may use a bitwise addition structure (Add) to perform bias processing on the second text feature to obtain an updated text feature with a bias characteristic. That is, on the second text feature, each bit is added with a specific value, such as, or other values, which are not limited here, and are only illustrated by taking "-1" as an example. Among them, the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature, such as "-1" in the updated text feature is used to identify the previous frame continuous frame text (business text), and the first text feature can be regarded as adding "0" to each bit to identify the current frame continuous frame text (target frame text). The frame timing relationship between the two features is identified by the size order of the numerical value "0" and the numerical value "-1". After determining the frame timing relationship between the updated text feature and the first text feature, the computer device can distinguish the frame timing relationship between the continuous frame texts corresponding to the two features, for example, the target frame text can be identified as the current frame text, and the business text can be identified as the historical frame text, so as to determine that the subsequent image generation process serves the current frame text. Alternatively, when the first text length threshold and the second text length threshold are subsequently determined based on the context window threshold, it can be determined that the first text length threshold corresponding to the current frame text is greater than the second text length threshold corresponding to the historical frame text, that is, the first text feature corresponding to the current frame text can account for a large proportion in the generation of the target text feature based on the first text feature and the updated text feature.
[0131] Further, the computer device generates a target text feature based on the first text feature and the updated text feature. Specifically, the computer device can obtain the context window threshold of the target video generation model, and determine the first text length threshold and the second text length threshold based on the context window threshold. Wherein, the sum of the first text length threshold and the second text length threshold is equal to the context window threshold. The computer device can generate a first spliced text feature based on the first text length threshold and the first text feature. Wherein, the feature length of the first spliced text feature is equal to the first text length threshold. The computer device can generate a second spliced text feature based on the second text length threshold and the updated text feature. Wherein, the feature length of the second spliced text feature is equal to the second text length threshold. The computer device can splice the first spliced text feature and the second spliced text feature to obtain the target text feature. For example, the feature lengths of the first text feature and the second text feature are n1 and n2 respectively, where n1 and n2 are positive integers. The computer device can obtain the context window threshold of the target video generation model (i.e., the feature length of the default input of the target video generation model, for example, 77). The computer device can determine that the first text length threshold is 38 and the second text length threshold is 37, i.e., the feature length of the generated first concatenated text feature is 38. When the feature length n1 of the first text feature is less than 38, the part less than 38 can be filled with the representation default value. When the feature length n2 of the second text feature is less than 37, the part less than 37 can also be filled with the representation default value. The first 37 bits in the target text feature can be the second concatenated text feature corresponding to the updated text feature, and the last 38 bits can be the first concatenated text feature corresponding to the first text feature. Among them, the target text feature has the same dimension as the target forward noise vector.
[0132] Furthermore, the computer device performs a back diffusion process on the target text feature and the target forward noise vector in the back diffusion network layer of the target video generation model to obtain the target prediction noise. The back diffusion network layer may include a noise prediction network, the noise prediction network includes a cross attention sub-network, and the computer device may perform a cross attention process on the target text feature and the target forward noise vector through the cross attention sub-network in the noise prediction network to obtain a cross attention process result, and determine the target prediction noise based on the cross attention process result.
[0133] Optionally, the noise prediction network may include D cross-attention subnetworks, where D is a positive integer. For example, the D cross-attention subnetworks include cross-attention subnetwork 1, cross-attention subnetwork 2, ..., cross-attention subnetwork D. In the cross-attention subnetwork 1, the computer device may use the target text feature as the key vector and value vector of the cross-attention function, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, the value vector and the query vector are cross-attention calculated to obtain the cross-attention processing result 1b. Further, the computer device may determine the cross-attention processing result 1b as the query vector in the cross-attention subnetwork 2, use the target text feature as the key vector and value vector of the cross-attention function, and obtain the cross-attention processing result 2b based on the cross-attention subnetwork 2, and so on, until the cross-attention processing of the D cross-attention subnetworks is completed, and the cross-attention processing result Db is obtained. Based on the cross-attention processing result Db, the target prediction noise is determined. It should be noted that the key vectors and value vectors in the D cross-attention subnetworks are both target text features.
[0134] The computer device can determine the target latent vector based on the target prediction noise and the target forward noise vector, that is, the computer device denoises the target forward noise vector through the target prediction noise to obtain the target latent vector. The computer device can reconstruct the target latent vector through the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and use the target prediction image as the next frame image located in the business image in the video to be generated. Further, the computer device obtains the next continuous frame text corresponding to the target frame text in P continuous frame texts, determines the next continuous frame text as the new target frame text, and generates a new target prediction image corresponding to the new target frame text based on the target video generation model. When the P continuous frame texts all pass through the target prediction image corresponding to the target video generation model, that is, P frame images are obtained, the computer device can determine the video to be generated containing P frame images as the business video. Among them, the target latent vector is used to characterize the characteristics of the target prediction image, that is, the potential representation of the target prediction image after noise removal in the latent space.
[0135] Through the above process, the target noise image corresponding to the target frame text can be determined based on the business image, so that the features of the business image can play a role in the subsequent process of generating the target prediction image; make full use of the feature extraction and feature splicing methods to extract and splice the features of the target frame text and the historical text (business text), and use the business text and the target frame text to jointly constrain and guide the target video generation model to generate a target prediction image for the target frame text, so that the features of the business text can play a role in the process of generating the target prediction image, thereby ensuring that the main objects of the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) can be kept consistent; by generating a target noise image based on the business image, a target prediction image corresponding to the target frame text is generated, thereby ensuring that the object scenes of the generated target prediction image and the historical image can be kept consistent. The similarity between the target predicted image and the historical image is improved, thereby improving the coherence and coordination of the video to be generated based on the target text; at the same time, the historical image constrains the target noise characteristics of the target noise sample, and the historical text further constrains the target video generation model to generate the target predicted image, which can cause the background (object scene), generation style or content of the target predicted image to be biased towards the historical image during generation, thereby improving the overall background similarity of the cultural video (referring to the video to be generated or business video generated based on the target text) (that is, the overall tone remains consistent), further ensuring the quality of the cultural video.
[0136] For further information, see Figure 4 , Figure 4 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 2 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S401-S407:
[0137] Step S401, obtain the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer.
[0138] In the embodiment of the present application, the specific implementation process of step S401 can be found in Figure 3 The specific description process in step S301 in will not be repeated here.
[0139] Step S402, obtain the target frame text traversed in order from P consecutive frame texts, obtain a noise data pool, obtain a target splicing noise map associated with the target frame text through the noise data pool, and generate a target noise image associated with the target frame text according to the low-frequency signal corresponding to the business image and the high-frequency signal corresponding to the target splicing noise map; the business image is the last frame image in the video to be generated, and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P consecutive frame texts.
[0140] In an embodiment of the present application, a computer device obtains a target frame text that is traversed in order from P consecutive frame texts, that is, the computer device performs image generation processing on the consecutive frame texts one by one in the order of the P consecutive frame texts. The computer device can obtain a noise data pool; the noise data pool includes S noise data; S is a positive integer. Among them, the noise data can be strip noise data. The computer device can obtain a sliding step length, and obtain P noise data groups from the S noise data through the sliding step length.
[0141] For details, please refer to Figure 5 , Figure 5 is a schematic diagram of a noise image acquisition scenario provided by an embodiment of the present application, such as Figure 5 As shown, the computer device can obtain P noise data groups from the S noise data in the noise data pool 501, and the number of noise data included in each noise data group is M (for example, Figure 5 Where M is 3 and the sliding step is 1). When the first noise data group in the P noise data groups is composed of noise data 1, noise data 2, and noise data 3 in the noise data pool, the second noise data group in the P noise data groups may be composed of noise data 2, noise data 3, and noise data 4 in the noise data pool. The computer device splices the M noise data in the P noise data groups respectively to obtain P second spliced noise images; the computer device determines a target spliced noise image associated with the target frame text from the P second spliced noise images.
[0142] For example, when the target frame text is the first continuous frame text (continuous frame text A1) among P continuous frame texts, the target splicing noise map associated with the target frame text may be the first second splicing noise map among the P second splicing noise maps. At this time, the target frame text does not have the previous continuous frame text, that is, there is no business text, and there is no business image, or in other words, the business text is blank null value text and the business image is blank null value image. The computer device may directly determine the first second splicing noise map as noise image 1, that is, the target noise image associated with the target frame text (continuous frame text A1 at this time). When the target frame text is the second continuous frame text (continuous frame text A2) among P continuous frame texts, the target splicing noise map associated with the target frame text may be the second second splicing noise map among the P second splicing noise maps. At this time, the computer device performs Fourier transform on the business image to obtain the first frequency domain corresponding to the business image, and the process of determining the frequency signal less than the low-frequency threshold in the first frequency domain as the low-frequency signal can be: the computer device performs Fourier transform on the business image 1 to obtain the first frequency domain corresponding to the business image 1 (the frame image obtained by the computer device performing image generation processing on the continuous frame text A1), and the frequency signal less than the low-frequency threshold in the first frequency domain is determined as the low-frequency signal 1d. The computer device performs Fourier transform on the target splicing noise image (the second second splicing noise image at this time) to obtain the target frequency domain corresponding to the second second splicing noise image, and determines the frequency signal greater than or equal to the low-frequency threshold in the target frequency domain as the high-frequency signal 1e. The computer device combines the high-frequency signal 1e with the low-frequency signal 1d to obtain a full-frequency signal, and performs inverse Fourier transform on the full-frequency signal to obtain a noise image 2f, that is, a target noise image associated with the target frame text (the continuous frame text A2 at this time). Similarly, when the target frame text is the third continuous frame text (continuous frame text A3) among P continuous frame texts, the target splicing noise image associated with the target frame text can be the third second splicing noise image among P second splicing noise images. At this time, the computer device can obtain the noise image 3f based on the business image 2 (the frame image obtained by the computer device performing image generation processing on the continuous frame text A2) and the high-frequency signal 2e in the target splicing noise image (the third second splicing noise image at this time), that is, the target noise image associated with the target frame text (the continuous frame text A3 at this time). Finally, when the P continuous frame texts are respectively used as the target frame text, the target noise images associated with them can be obtained, that is, P target noise images, or P noise images. The noise image 1f among the P noise images is the target noise image associated with the target frame text when it is the continuous frame text A1 (the first second splicing noise image at this time).
[0143] Among them, the Fourier transform can be a 3D Fourier transform (3DFT), through which a three-dimensional function or signal can be represented as the sum of its component frequencies. For images, it means converting information in the spatial domain to the frequency domain. In the frequency domain, low-frequency signals are usually concentrated in the low-frequency part of the transform result, that is, the area close to the origin. The computer device can extract these low-frequency components (i.e., low-frequency signals) by setting a low-frequency threshold or using a low-pass filter. The specific value of the low-frequency threshold can be determined by the video generation task corresponding to the target text. For example, if the generation task corresponding to the target text is a scene video generation task, the low-frequency threshold can be determined as 5 Hz, and the global low-frequency signal is a frequency signal with a frequency less than 5 Hz; the low-frequency threshold of the person or pet video generation task can be determined as 10 Hz. The inverse Fourier transform can be a 3D inverse Fourier transform (3D IFFT), which is used to convert a signal or function from the frequency domain back to its original time domain or space domain. In the one-dimensional case, the inverse Fourier transform is the process of converting frequency domain data back to the time domain; in the multi-dimensional case, such as 3D IFFT, the inverse Fourier transform converts the three-dimensional frequency domain data back to the three-dimensional space domain. The noise data pool has overlapping sampling, and the second spliced noise image is obtained, and the low-frequency signal of the latest frame image (business image) and the high-frequency signal of the second spliced noise image are combined to obtain the target noise image associated with the target frame text. This method of initializing noise can ensure the overall background similarity of the Wensheng video.
[0144] Step S403, input the target noise image into the target video generation model, extract features of the target noise image through the target video generation model to obtain target noise features, perform forward diffusion processing on the target noise features to obtain a target forward noise vector.
[0145] In the embodiment of the present application, the specific implementation process of step S403 can be found in Figure 3 The specific description process of step S303 in is not repeated here. Figure 6 , Figure 6 is a schematic diagram of a target video generation model structure provided in an embodiment of the present application, such as Figure 6 As shown, the computer device inputs the target noise image obtained based on the business image and the noise data pool into the target video generation model, and extracts the features of the target noise image through the target video generation model to obtain the target noise feature Z. The target noise feature Z is further forward diffused for T time steps (time step length) through the forward diffusion network layer FD to obtain the target forward noise vector Z output by the forward diffusion network layer FD.T Among them, forward diffusion can refer to the process of adding noise to the target noise feature Z, and reverse diffusion can refer to the process of adding noise to the target forward noise vector Z. T The time step is used to indicate the amplitude of the added noise in the forward diffusion and the amplitude of the predicted noise in the backward diffusion.
[0146] Step S404, inputting the business text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the business text to obtain a second text feature.
[0147] In the embodiment of the present application, the specific implementation process of step S404 can be found in Figure 3 The specific description process of obtaining the text encoding feature in step S304 in the text encoding feature is not repeated here. Figure 6 ,like Figure 6 As shown, the computer device inputs the business text and the target frame text into the target video generation model, and performs text encoding on the target frame text through the text encoding layer τ to obtain the first text feature; and performs text encoding on the business text through the text encoding layer τ to obtain the second text feature. Wherein, the business text is a continuous frame text used to characterize the business image in P continuous frame texts, and the business text is the previous continuous frame text of the target frame text. Wherein, the text encoding layer τ can be a text encoder (TextEncoder) in the contrastive language-image pre-training model (Contrastive Language-Image Pre-Training, CLIP). Taking the business text as an example, the computer device can perform text segmentation on the business text through the Text Encoder to obtain a word unit sequence corresponding to the business text (including G words, G is a positive integer), and perform vector conversion on each word unit in the word unit sequence corresponding to the business text to obtain the word embedding vectors corresponding to the G words, and add text position information to each word embedding vector to obtain G word update vectors. Wherein, the text position information refers to the information of the position of a word unit in the business text. Perform feature fusion on the G word update vectors to obtain the second text feature.
[0148] Step S405, bias processing is performed on the second text feature to obtain an updated text feature with a bias characteristic; the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature; based on the first text feature and the updated text feature, a target text feature is generated.
[0149] In the embodiment of the present application, the specific implementation process of step S405 can be found in Figure 3The specific description process of obtaining the target text features in step S305 is not repeated here. Figure 6 ,like Figure 6 As shown, the computer device performs bias processing on the second text feature through the bitwise addition structure Add to obtain an updated text feature with a bias characteristic, and then generates a target text feature based on the first text feature and the updated text feature.
[0150] Step S406, in the back diffusion network layer in the target video generation model, based on the target text features, the target forward noise vector is back diffused in T time steps to obtain the target prediction noise; T is a positive integer; based on the target prediction noise and the target forward noise vector, the target latent vector is determined; the target latent vector is image reconstructed by the decoder in the target video generation model to obtain the target prediction image corresponding to the target frame text, and the target prediction image is used as the next frame image located in the business image in the video to be generated.
[0151] In the embodiment of the present application, T time steps include time step T i , i is a positive integer less than or equal to T, and the computer device can be in the time step T of the back diffusion network layer in the target video generation model i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector.
[0152] Specifically, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each of which includes a cross-attention sub-network. When T is 3, the T noise prediction networks may include noise prediction network 1, noise prediction network 2, and noise prediction network 3. Noise prediction network 1, noise prediction network 2, and noise prediction network 3 each include a cross-attention sub-network. The computer device at time step T i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The specific implementation process of the corresponding initial noise vector can be: the computer device at time step T i In the example, the target text features and iterative noise vector are input to the time step T i The corresponding cross attention sub-network. Through time step T iThe corresponding cross-attention sub-network in the noise prediction network performs cross-attention processing on the target text features and the iterative noise vector, and obtains the cross-attention processing result. Specifically, the computer device can use the initial noise vector as the query vector in the cross-attention function, the target text features as the key vector in the cross-attention function, and the target text features as the value vector in the cross-attention function. The computer device determines the product of the query vector and the key vector as the first fused feature through the cross-attention function, obtains the number of dimensions corresponding to the target text features, determines the product of the first fused feature and the inverse of the number of dimensions as the second fused feature, and converts the second fused feature into the first activated feature based on the activation sub-function in the cross-attention function, and determines the product of the first activated feature and the value vector as the cross-attention processing result. Among them, a possible cross-attention function can be seen in formula ①:
[0153]
[0154] As shown in formula ①, Attention(Q,K,V) represents the cross attention function, Q is used to represent the query vector query, K is used to represent the key vector key, and V is used to represent the value vector value; T represents transposition, K T represents the transposed matrix of K, d k Used to represent the number of dimensions corresponding to the target text features; is the first activated feature, The result of cross attention processing.
[0155] Further, the computer device may determine the time step T based on the cross attention processing result. i The computer device can convert the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the above example, based on the fusion coding features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise. i+1 is the time step T among T time steps i The next time step.
[0156] It should be noted that the noise prediction network corresponding to each time step can include multiple cross-attention sub-networks, please refer to Figure 6 ,like Figure 6As shown, the target video generation model includes a back diffusion network layer BD, and the computer device generates a target forward noise vector Z through the back diffusion network layer BD. T Perform back diffusion processing (denoising processing) for T time steps to obtain the target prediction noise Z0 output by the back diffusion network layer BD. In the first time step (i.e., time step T1), the noise prediction network ∈ 1 may include multiple cross-attention sub-networks. For example, the noise prediction network ∈ 1 may include D (D may be equal to 16) cross-attention sub-networks (Note: Figure 6 Only four cross-attention sub-networks are drawn, including cross-attention sub-network U1, cross-attention sub-network U2, cross-attention sub-network U3, and cross-attention sub-network U4). Taking the four cross-attention sub-networks as an example, the computer device can use the target text feature as the key vector and value vector of the cross-attention function in the cross-attention sub-network U1, and use the target forward noise vector as the query vector of the cross-attention function. Through the cross-attention function, the key vector, value vector and query vector are cross-attention calculated to obtain the cross-attention processing result 1a. Furthermore, the computer device can determine the cross-attention processing result 1a as the query vector in the cross-attention sub-network U2, use the target text feature as the key vector and value vector of the cross-attention function, and obtain the cross-attention processing result 2a based on the cross-attention sub-network 2. And so on, until the processing of D (for example, 4) cross-attention sub-networks is completed, and the cross-attention processing result Da is obtained. It should be noted that the key vectors and value vectors in the D cross-attention sub-networks are all target text features. The computer device determines the initial noise vector Z corresponding to the first time step based on the cross-attention processing result Da. T-1 . Further, in the remaining T-1 time steps, continue to use the noise prediction network ∈ i The initial noise vector Z corresponding to the first time step T-1 Perform reverse diffusion processing until the target prediction noise Z0 is obtained.
[0157] Among them, the noise prediction network includes several downsampling network layers, one intermediate sampling network layer and several upsampling network layers, such as Figure 6 The cross-attention sub-network U1 and the cross-attention sub-network U2 shown belong to the downsampling network layer, such as Figure 6 The cross-attention sub-network U3 and cross-attention sub-network U4 shown belong to the upsampling network layer. Please also refer to Figure 7 , Figure 7 is a schematic diagram of a downsampling network layer structure provided in an embodiment of the present application, such as Figure 7As shown, the downsampling network layer includes 2 residual processing layers (ResnetBlock2D), 2 transform processing layers (Transformer2DModel) and 1 downsampling processing unit (DownSample2D). The computer device converts the target forward noise vector Z T , target text features and time embedding vector (used to characterize the number of time steps, here taking the first time step as an example), input into the first residual processing layer in the downsampling network layer, obtain the output result of the first residual processing layer, further input the output result into the first transformation processing layer, obtain the output result of the first transformation processing layer, target text features and time embedding vector, input into the second residual processing layer, obtain the output result of the second residual processing layer and target text features, and input into the second transformation processing layer, input the input result of the second transformation processing layer into the downsampling processing unit. The output result of the downsampling processing unit (i.e., the output result of the downsampling processing network) is used to input into the next downsampling processing network for subsequent data processing. Among them, the main function of the residual processing layer is to help the downsampling network layer solve the gradient vanishing problem, especially in the deep network, by adding residual connections, the downsampling network layer can more easily learn the identity mapping, that is, directly pass the input to the output without losing information in the deep network. A typical ResnetBlock2D contains two convolutional layers, each of which is usually followed by batch normalization and ReLU activation functions. After the input of ResnetBlock2D passes through these layers, its output is added to the input to form a residual connection. In the image segmentation task, Transformer2DModel allows the target video generation model to consider global context information when processing images through the self-attention mechanism. In the noise prediction network (U-net), Transformer2DModel can be used as part of the encoder or decoder, or as a bridge between the two to provide richer feature representation and context information. DownSample2D is used to reduce the spatial dimension of the input data, that is, the height and width of the image, while possibly increasing the number of channels.
[0158] Please also refer to Figure 8 , Figure 8 is a schematic diagram of a transformation processing layer structure provided in an embodiment of the present application, such as Figure 8 As shown in Figure 1, the change processing layer includes 1 normalization processing layer (GroupNorm), 2 convolution processing layers (Conv1×1), 2 exchange processing layers and 1 transformation processing unit (BasicTransformerBlock). The input of the transformation processing layer can be as follows: Figure 7The output of the residual processing layer shown. The input of the transformation processing unit is the target text feature and the output of the first exchange processing layer. The normalization processing layer (GroupNorm) is used to divide the channels corresponding to each input feature map (noise image) into multiple groups, and then normalize them within each group. Conv1×1 convolution is usually used to reduce or increase the number of channels of the feature map without changing the spatial dimensions (height and width) of the feature map. The exchange processing layer includes a permute function and a reshape operation, where the permute function is used to rearrange the dimensions of the tensor (feature or vector). For example, a tensor of shape (N, C, H, W), where N is the batch size, C is the number of channels, H is the height, and W is the width, can be rearranged to (N, H, W, C) using permute. This rearrangement can convert image data from a two-dimensional spatial feature map into a serialized form so that it can be input into the transformation processing unit, which is useful. The reshape operation is used to change the shape of a tensor without changing its data. In the transformation processing layer, the reshape operation is often used to convert the serialized feature map back to its original 2D or 3D shape for further processing or combining with other parts of the U-Net.
[0159] The structure of the transformation processing unit can be found in Fig. 9 , Fig. 9 is a schematic diagram of a transformation processing unit structure provided in an embodiment of the present application, such as Fig. 9 As shown in Figure 1, the transform processing unit includes three linear processing layers (LayerNorm), a self-attention processing layer (SelfAttention), a cross-attention processing layer (CrossAttention) and a feedforward processing layer. The input of the transform processing unit can be as follows Figure 8 The output of the first exchange processing layer shown. Among them, the linear processing layer is used to perform a linear transformation on the input to obtain a linear output. The self-attention processing layer is used to capture the relationship between any two elements in the input sequence. The cross-attention processing layer is used to capture the attention relationship between two sequences (the target text feature and the output of the second linear processing layer), that is, the attention relationship between the target forward noise vector or the iterative noise vector and the target text feature. The computer device can input the cross-attention processing result obtained by the cross-attention processing layer into the third linear processing layer of the transformation processing unit, and input the output result of the linear processing layer into the feedforward processing layer. The feedforward processing layer can be used to perform feature transformation on the input sequence, introduce a nonlinear activation function (for example, a ReLU function), and perform nonlinear processing.
[0160] Further, the computer device predicts the noise Z0 and the target forward noise vector Z TDetermine the target latent vector, reconstruct the target latent vector through the decoder D in the target video generation model, and obtain the target prediction image Y corresponding to the target frame text. Use the target prediction image Y as the next frame image located in the business image in the video to be generated. Furthermore, the computer device obtains the next continuous frame text corresponding to the target frame text in P continuous frame texts, determines the next continuous frame text as the new target frame text, and generates a new target prediction image corresponding to the new target frame text based on the target video generation model. When the P continuous frame texts all pass through the target prediction image corresponding to the target video generation model, that is, P frame images are obtained, the computer device can obtain a video to be generated including P frame images, wherein the P frame images include the target prediction image Y.
[0161] Step S407: interpolate frames between every two adjacent frames of the video to be generated.
[0162] In an embodiment of the present application, after obtaining the video to be generated, the computer device may also perform interpolation processing between every two adjacent frame images of the video to be generated, so as to further increase the coherence of the video to be generated. Specifically, when the video to be generated contains frame images corresponding to P consecutive frame texts, the computer device performs image interpolation on every two adjacent frame images to obtain an intermediate frame image between every two adjacent frame images. The frame images corresponding to the P consecutive frame texts include the target prediction image. The frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P. The computer device can be based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result. j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image.
[0163] Specifically, the computer device can use the optical flow algorithm to calculate the frame image P j-1 To frame image P j The optical flow vector field and frame image Pj+1 To frame image P j+2 The optical flow vector field is used to describe the movement of pixels in space. For example, the frame image P j-1 To frame image P j The optical flow vector field is used to describe the frame image P j-1 The pixel points in the frame image P j-1 The corresponding time to frame image P j The corresponding moment, the operation in space. The computer device can use the quadratic optical flow model based on the frame image P j-1 To frame image P j The optical flow vector field and frame image P j+1 To frame image P j+2 The optical flow vector field is calculated from the pixel point of the frame image P j The corresponding time to frame image P j+1 The optical flow vector field of a certain moment in the corresponding moment is calculated, and the position of the pixel corresponding to that moment is predicted (i.e., the pixel position of the intermediate frame). Further, the computer device uses a differentiable optical flow reversal layer to predict the reverse optical flow, that is, from the intermediate frame back to the frame image P j and from the intermediate frame to the frame image P j+1 The optical flow vector field (i.e., the optical flow prediction result) of the frame image P j , frame image P j+1 And the optical flow prediction results, through image reprojection and interpolation technology, generate the frame image P j and frame image P j+1 Optionally, the computer device may also apply an adaptive flow filter to process the inverted optical flow (reverse optical flow) to reduce possible ringing effects and improve the quality of the intermediate frames. Further, the computer device determines the video to be generated, which includes the frame images corresponding to the P consecutive frame texts and the intermediate frame images, as a business video matching the target text.
[0164] Through the above process, the target noise image corresponding to the target frame text can be determined based on the business image, so that some information of the business image (low-frequency signals related to the image background) can be present in the target noise image for generating the target prediction image, thereby ensuring that the target prediction image finally generated and the business image have some identical features; make full use of the feature extraction and feature splicing methods to extract and splice the features of the target frame text and the historical text (business text), and generate a target prediction image for the target frame text by jointly constraining and guiding the target video generation model through the business text and the target frame text, so that the features of the business text can play a role in the process of generating the target prediction image, thereby ensuring that the main objects of the generated target prediction image and the historical image (i.e., the previous and next frames of the video to be generated) can be kept consistent; generate the target prediction image corresponding to the target frame text by generating the target noise image based on the business image, thereby ensuring that the object scenes of the generated target prediction image and the historical image can be kept consistent. The similarity between the target predicted image and the historical image is improved, thereby improving the coherence and coordination of the video to be generated based on the target text; at the same time, the historical image constrains the target noise characteristics of the target noise sample, and the historical text further constrains the target video generation model to generate the target predicted image, which can cause the background (object scene), generation style or content of the target predicted image to be biased towards the historical image during generation, thereby improving the overall background similarity of the Wensheng video (referring to the video to be generated or the business video generated based on the target text) (that is, the overall tone remains consistent); and an intermediate frame image can be inserted between every two adjacent frame images in the video to be generated, further improving the coherence of the business video indicated by the target text, and further ensuring the quality of the Wensheng video.
[0165] See also Fig.10 , Fig.10 This is a flow diagram of a data processing method provided in an embodiment of the present application. Figure 3 The data processing method can be executed by a computer device, which can be Figure 1 The business server 101 shown or any terminal device in the terminal device cluster. The following will take the data processing method executed by a computer device as an example for explanation. The data processing method may at least include the following steps S1001-S1007:
[0166] Step S1001, obtaining N image samples in a video frame sorting order, and a text sample respectively associated with each image sample; N is a positive integer.
[0167] In an embodiment of the present application, a computer device may obtain a sample video, obtain N video frames based on the sorting order of video frames in the sample video, and determine the N video frames as N image samples. Wherein, N is a positive integer, the N image samples all contain the same subject object and object scene, and the display perspectives of the subject objects in the N image samples are different from each other. The computer device may generate text samples for the N image samples respectively through an image description generation model to obtain N text samples. Wherein, the text sample is used to indicate the specific content in the corresponding image sample. The image description generation model may be a generative adversarial network model (Generative Adversarial Networks, GAN), a BLIP model or a scholar multimodal large model (InternVL), or other models that can be used as image generation text descriptions, which are not limited here.
[0168] Step S1002, obtaining a noise data pool, and determining N noise samples based on the noise data pool and the N image samples; one noise sample is determined by the noise data pool and one image sample.
[0169] In an embodiment of the present application, a computer device may obtain a noise data pool, wherein the noise data pool includes S noise data, where S is a positive integer. The computer device may obtain a sample sliding step length, and obtain N sample noise data groups from the S noise data through the sample sliding step length, and splice the noise data in the N noise data groups respectively to obtain N first sample noise maps; the number of noise data included in each sample noise data group is R, where R is a positive integer. The computer device may generate N noise samples based on the N first sample noise maps and N image samples.
[0170] Each image sample corresponds to one of the N first sample noise maps, and the N image samples include image samples C i , N first sample noise images include image sample C i The corresponding first sample noise map B i , i is a positive integer less than or equal to N. Please also refer to Fig.11 , Fig.11 is a schematic diagram of a noise sample acquisition scenario provided in an embodiment of the present application, such as Fig.11 As shown, the sample sliding step is 1, R is 3, and the computer device extracts the noise data from the noise data pool 1101 (which can be compared with Figure 5 ), obtain N sample noise data groups from the noise data pool 501 shown in , respectively splice the noise data in the N noise data groups, and obtain N first sample noise graphs (including the first sample noise graph B1, the first sample noise graph B2, the first sample noise graph B3, ..., the first sample noise graph B N). The number of noise data included in each sample noise data group is 3. The N image samples include image sample C1, image sample C2, image sample C3, image sample C N-1 , image sample C N Among them, the image sample C1 corresponds to the first first sample noise map (first sample noise map B1) of the N first sample noise maps, the image sample C2 corresponds to the first sample noise map B2, the image sample C3 corresponds to the first sample noise map B3, and the image sample C N-1 Corresponding to the first sample noise map B N-1 , image sample C N Corresponding to the first sample noise map B N .
[0171] Specifically, when the image sample C i When the first ranked image sample among the N image samples is selected, the computer device may generate the first sample noise map B i , determined as image sample C i The corresponding noise sample. That is, image sample C i When it is the image sample C1, the computer device may determine the first sample noise map B1 as the noise sample corresponding to the image sample C1, that is, the noise sample 1.
[0172] Optionally, when image sample C i When the image sample C is not the first image sample in the N image samples, the computer device can i-1 Perform Fourier transform to obtain image sample C i-1 The first sample frequency domain corresponding to the first sample frequency domain determines the frequency signal less than the sample low-frequency threshold in the first sample frequency domain as the sample low-frequency signal, and the first sample noise map B i Perform Fourier transform to obtain the first sample noise image B i The corresponding second sample frequency domain, the frequency signal greater than or equal to the sample low-frequency threshold in the second sample frequency domain is determined as the sample high-frequency signal, the sample high-frequency signal is combined with the sample low-frequency signal to obtain a sample full-frequency signal, and the sample full-frequency signal is inverse Fourier transformed to obtain an image sample C i The corresponding noise sample. Among them, the image sample C i-1 is the image sample C among N image samples i For example, when image sample C iWhen the image sample is C2, the computer device can perform Fourier transform on the image sample C1 to obtain the first sample frequency domain 1 corresponding to the image sample C1, determine the frequency signal 1 less than the sample low-frequency threshold in the first sample frequency domain 1 as the sample low-frequency signal 1, perform Fourier transform on the first sample noise map B2 to obtain the second sample frequency domain 1 corresponding to the first sample noise map B2, determine the frequency signal greater than or equal to the sample low-frequency threshold in the second sample frequency domain 1 as the sample high-frequency signal 1, combine the sample high-frequency signal 1 with the sample low-frequency signal 1 to obtain the sample full-frequency signal 1, perform inverse Fourier transform on the sample full-frequency signal 1 to obtain the noise sample corresponding to the image sample C2, that is, the noise sample 2. Similarly, the computer device can determine the noise sample 3 corresponding to the image sample C3 based on the image sample C2 and the first sample noise map B3; the computer device can determine the noise sample 3 corresponding to the image sample C3 based on the image sample C2 and the first sample noise map B3; the computer device can determine the noise sample 3 corresponding to the image sample C3 based on the image sample C N-1 and the first sample noise map B N Determine the image sample C N The corresponding noise samples N are obtained, thereby obtaining the noise samples corresponding to the N image samples respectively (ie, N noise samples).
[0173] Step S1003: determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples.
[0174] In the embodiment of the present application, the N image samples may include image1 to imageN; the N text samples may include text1 to textN. The image samples and the text samples correspond one to one, for example, image1 corresponds to text1, and imageN corresponds to textN. The first target image sample determined by the computer device from the N image samples is image1, and the target text sample associated with the target image sample determined from the N text samples is text1.
[0175] Step S1004, feature extraction is performed on the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain a potential noise feature, and forward diffusion processing is performed on the potential noise feature to obtain a sample forward noise vector; the target noise sample is determined by the first image sample; the first image sample is the previous image sample of the target image sample in the N image samples.
[0176] In the embodiment of the present application, the computer device can determine the target noise sample corresponding to the target image sample from the N noise samples. For example, if the target image sample is image1, the target noise sample can be the first noise sample (noise sample 1) among the N noise samples. Fig.12 , Fig.12 is a schematic diagram of an initial video generation model structure provided in an embodiment of the present application, such as Fig.12 As shown, the computer device can input the target noise sample and the target image sample x into the initial video generation model, extract the target image sample features through the encoder E (for example, it can be an image encoder VAE) in the initial video generation model, and obtain the target image sample features. The target noise sample is extracted through the initial video generation model to obtain a target noise sample vector, and the latent noise feature Z' is determined based on the target image sample features and the target noise sample vector. In the T time steps of the initial forward diffusion network layer FD, the computer device continuously adds the noise vector (target noise sample vector) corresponding to the target noise sample to the latent noise feature to obtain the sample forward noise vector Z T '. The target noise sample is determined by the first image sample; the first image sample is the previous image sample of the target image sample in the N image samples. That is, when the target image sample is image sample C2, the first image sample is image sample C1, and the target noise sample is noise sample 1 corresponding to image sample C1.
[0177] Step S1005, text encoding is performed on the target text sample to obtain a first encoding feature, and text encoding is performed on the first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample.
[0178] In the embodiment of the present application, the specific implementation process of the computer device performing text encoding on the target text sample to obtain the first encoding feature, and the specific implementation process of performing text encoding on the first text sample to obtain the second encoding feature, can be referred to as Figure 3 Step S304 shown in , or as Figure 4 In step S404 shown in , the business text is encoded to obtain a specific description of the second text feature, which will not be repeated here. Fig.12 ,like Fig.12 As shown, the initial video generation model includes a text encoder τ, and the computer device can use the text encoder τ to perform text encoding on the target text sample to obtain a first encoding feature; the computer device can use the text encoder τ to perform text encoding on the first text sample to obtain a second encoding feature.
[0179] Step S1006, performing reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector.
[0180] In the embodiment of the present application, the computer device can perform feature concatenation on the first coding feature and the second coding feature to obtain the target coding feature, and perform reverse diffusion processing on the sample forward noise vector and the target coding feature to obtain the predicted noise sample vector. The specific implementation process of the computer device performing feature concatenation on the first coding feature and the second coding feature to obtain the target coding feature can be referred to as Figure 3 In step S305, the first text feature and the second text feature are concatenated to obtain a specific description of the target text feature, which will not be described in detail here. Fig.12 ,like Fig.12 As shown, the initial video generation model includes a bitwise addition structure Add, and the computer device can bias the second coding feature through the bitwise addition structure Add to obtain an updated coding feature with a bias characteristic, and generate a target coding feature based on the updated coding feature. The initial video generation model also includes an initial back diffusion network layer BD, and the initial back diffusion network layer BD includes an initial noise prediction network (each time step corresponds to an initial noise prediction network), and each initial noise prediction network includes multiple cross-attention sub-networks. For example, in the first time step (i.e., time step T1), the initial noise prediction network ∈ 1 may include D (D may be equal to 16) initial cross-attention sub-networks (Note: Fig.12 Only four initial cross-attention sub-networks are drawn, namely initial cross-attention sub-network O1, initial cross-attention sub-network O2, initial cross-attention sub-network O3, and initial cross-attention sub-network O4). The computer device can use the target encoding feature as the key vector and value vector of the cross-attention function in the initial cross-attention sub-network O1, and use the sample forward noise vector Z T 'As the query vector of the cross-attention function, the key vector, value vector and query vector are cross-attention calculated through the cross-attention function to obtain the sample attention processing result 1c. The computer device can determine the sample attention processing result 1c as the query vector in the initial cross-attention sub-network O2, and use the target encoding feature as the key vector and value vector of the cross-attention function, and obtain the sample attention processing result 2c based on the initial cross-attention sub-network O2. And so on, until the processing of D (for example, 16) initial cross-attention sub-networks is completed, and the sample attention processing result Dc is obtained. It should be noted that the key vectors and value vectors in the D initial cross-attention sub-networks are all target encoding features. The computer device determines the initial sample noise vector Z corresponding to the first time step based on the sample attention processing result Dc. T-1 '. Further, in the remaining T-1 time steps, the computer device continues to use the initial noise prediction network ∈ i For the initial sample noise vector Z corresponding to the first time step T-1'Perform reverse diffusion processing until the predicted noise sample vector Z0' is obtained.
[0181] Step S1007, adjusting the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.
[0182] In an embodiment of the present application, the initial video generation model includes an initial reverse diffusion network layer, which is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature, and the second coding feature to obtain a predicted noise sample vector. After obtaining the predicted noise sample vectors corresponding to the N noise samples, the computer device can generate a model loss value based on the N noise samples and the N predicted noise sample vectors. The generation process of the model loss value can refer to formula ②:
[0183]
[0184] As shown in formula ②, L LDM It is used to represent the model loss value, E represents the expected value, ε(x) is used to represent the encoding result of the encoder E in the initial video generation model encoding the target image sample x, y is used to represent the initial video generation model, based on the target image sample x, the target sample prediction image y is obtained, ∈~Ν(0,1), and is used to represent that the random noise ∈ obeys the standard normal distribution, that is, the Gaussian distribution with a mean of 0 and a variance of 1. In the embodiment of the present application, the random noise ∈ is the target noise sample. t is used to represent the time step, ∈ θ The predicted noise sample vector for the target noise sample output by the initial video generation model, z t Used to represent the sample forward noise vector (i.e. Fig.12 Z T '), c is used to represent the target encoding feature. represents the noise ∈ predicted by the initial video generation model θ The L2 loss between (predicted noise sample vector) and the actual sampled noise ∈ (target noise sample). This loss measures the initial video generation model given the sample forward noise vector z t , time step t, and target encoding feature c, the accuracy of noise prediction. Among them, L2 loss is also called Euclidean loss or mean squared error (MSE), which is used to measure the sum of squares of the difference between the model prediction value and the actual observation value (true value), that is, the sum of squares of the difference between the target noise sample (actual observation value) and the predicted noise sample vector (model prediction value) of the initial video generation model.
[0185] Furthermore, the computer device can adjust the parameters of the initial back diffusion network layer based on the model loss value to obtain a target video generation model. The target video model is used to generate a target video indicated by the target text based on the target text. Optionally, in the process of training the initial video generation model, the computer device can use multiple batches of noise samples, image samples and text samples, set a certain learning rate (for example, 1e-4), and perform multiple rounds of training on the initial video generation model. When each round of training is completed, the model loss value corresponding to the round of training is used to adjust the parameters of the initial video generation model until multiple rounds of training are completed to obtain the final target video generation model.
[0186] Through the above process, the initial video generation model can learn how to obtain the predicted noise sample vector according to the sample forward noise vector and the target coding feature corresponding to the target noise sample, so as to generate an image similar to the target image sample. Among them, the target noise sample is determined by the first image sample (historical sample image); the target coding feature is determined by the target text sample and the first text sample (historical sample text) as joint control information, so that the initial video generation model can learn the association relationship from the historical sample image to the current frame image (target image sample), so that the trained target video generation model has the ability to generate the content of the previous and next frames in a consistent manner, and ensures that the features corresponding to the business image and the business text can play a role when the target video generation model predicts the target predicted image. Make the target predicted image consistent and coherent with the business image.
[0187] Furthermore, the embodiment of the present application performs task adaptability training on the basis of the existing SD model, reuses the existing SD model, and transforms it on the basis of the existing SD model to obtain an initial video generation model, that is, the input of the SD model is changed on the existing SD model, that is, from being controlled by the target text sample to being jointly controlled by the target text sample and the first text sample; and the noise acquisition method of the SD model is changed, and the target noise sample is changed to be determined by the first image sample, so as to ensure that the first text sample and the first image sample play a role in the training of the initial video generation model, so that the initial video generation model has the ability to generate the content of the previous and next frames consistent, thereby avoiding the mismatch between the target video generation model and the target video generation task caused by non-training methods.
[0188] For further information, see Fig.13 , Fig.13 A data processing device 1300 provided in an embodiment of the present application is shown in FIG. Figure 1 The data processing device 1300 may include: a data acquisition module 11 , a data determination module 12 , a data forward diffusion module 13 , a feature encoding module 14 , an image prediction module 15 , and a video determination module 16 .
[0189] The data acquisition module 11 is used to acquire the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer;
[0190] The data determination module 12 is used to obtain the target frame text traversed in order from the P continuous frame texts, obtain the noise data pool, and determine the target noise image associated with the target frame text based on the business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is the frame image represented by the business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts;
[0191] The data forward diffusion module 13 is used to input the target noise image into the target video generation model, extract the features of the target noise image through the target video generation model to obtain the target noise features, and perform forward diffusion processing on the target noise features to obtain the target forward noise vector;
[0192] A feature encoding module 14 is used to input the business text and the target frame text into the target video generation model, perform text encoding on the target frame text through the target video generation model to obtain a first text feature, and perform text encoding on the business text to obtain a second text feature;
[0193] The image prediction module 15 is used to denoise the target noise image according to the target forward noise vector, the first text feature and the second text feature, obtain the target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located in the business image in the video to be generated.
[0194] In a possible implementation, when the data determination module 12 is used to determine the target noise image associated with the target frame text based on the business image and the noise data pool, the data determination module 12 is specifically used to perform the following operations:
[0195] A target splicing noise map associated with the target frame text is obtained through the noise data pool, and a target noise image associated with the target frame text is generated according to the low-frequency signal corresponding to the business image and the high-frequency signal corresponding to the target splicing noise map.
[0196] In a possible implementation, the noise data pool includes S noise data, where S is a positive integer; the data determination module 12 is used to obtain a target splicing noise image associated with the target frame text through the noise data pool, and when generating a target noise image associated with the target frame text according to a low-frequency signal corresponding to the business image and a high-frequency signal corresponding to the target splicing noise image, the data determination module 12 is specifically used to perform the following operations:
[0197] Obtain a sliding step length, obtain P noise data groups from the S noise data by means of the sliding step length, respectively splice the noise data in the P noise data groups to obtain P second spliced noise images, and determine a target spliced noise image associated with the target frame text from the P second spliced noise images; the number of noise data included in each noise data group is M, and M is a positive integer;
[0198] Performing Fourier transform on the business image to obtain a first frequency domain corresponding to the business image, and determining a frequency signal in the first frequency domain that is less than a low-frequency threshold as a low-frequency signal;
[0199] Performing Fourier transform on the target splicing noise image to obtain a target frequency domain corresponding to the target splicing noise image, and determining a frequency signal in the target frequency domain that is greater than or equal to a low-frequency threshold as a high-frequency signal;
[0200] The high-frequency signal is combined with the low-frequency signal to obtain a full-frequency signal, and the full-frequency signal is inversely Fourier transformed to obtain a target noise image associated with the target frame text.
[0201] In a possible implementation, the image prediction module 15 is used to perform denoising on the target noise image according to the target forward noise vector, the first text feature, and the second text feature. When obtaining the target predicted image corresponding to the target frame text, the image prediction module 15 is specifically used to perform the following operations:
[0202] Perform feature concatenation on the first text feature and the second text feature to obtain a target text feature;
[0203] In the reverse diffusion network layer in the target video generation model, based on the target text features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer;
[0204] Determine a target latent vector based on the target prediction noise and the target forward noise vector;
[0205] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.
[0206] In a possible implementation, the image prediction module 15 is used to perform feature concatenation on the first text feature and the second text feature to obtain the target text feature. The image prediction module 15 is specifically used to perform the following operations:
[0207] Performing bias processing on the second text feature to obtain an updated text feature with a bias characteristic; the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature;
[0208] A target text feature is generated based on the first text feature and the updated text feature.
[0209] In a possible implementation, when the image prediction module 15 is used to generate the target text feature based on the first text feature and the update text feature, the image prediction module 15 is specifically used to perform the following operations:
[0210] Obtaining a context window threshold of a target video generation model, and determining a first text length threshold and a second text length threshold based on the context window threshold; the sum of the first text length threshold and the second text length threshold is equal to the context window threshold;
[0211] Based on the first text length threshold and the first text feature, a first concatenated text feature is generated; the feature length of the first concatenated text feature is equal to the first text length threshold;
[0212] Based on the second text length threshold and the updated text feature, a second concatenated text feature is generated; the feature length of the second concatenated text feature is equal to the second text length threshold;
[0213] The first concatenated text feature and the second concatenated text feature are concatenated to obtain a target text feature.
[0214] In a possible implementation, the image prediction module 15 is used to perform denoising on the target noise image according to the target forward noise vector and the target text feature, and when obtaining the target predicted image corresponding to the target frame text, the image prediction module 15 is specifically used to perform the following operations:
[0215] In the reverse diffusion network layer in the target video generation model, based on the target text features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain the target prediction noise; T is a positive integer;
[0216] Determine a target latent vector based on the target prediction noise and the target forward noise vector;
[0217] The target latent vector is reconstructed through the decoder in the target video generation model to obtain the target predicted image corresponding to the target frame text.
[0218] In one possible implementation, the T time steps include time step T i , i is a positive integer less than or equal to T; the image prediction module 15 is used to perform reverse diffusion processing on the target forward noise vector in T time steps based on the target text feature, and when the target prediction noise is obtained, the image prediction module 15 is specifically used to perform the following operations:
[0219] At time step T i In the example, based on the target text features, the time step Ti The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step among T time steps, then time step T i The corresponding iterative noise vector is the target forward noise vector;
[0220] Set the time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector, at time step T i+1 In the example, based on the target text features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; time step T i+1 is the time step T among T time steps i The next time step.
[0221] In a possible implementation, the back diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network; the image prediction module 15 is used to i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i When the corresponding initial noise vector is , the image prediction module 15 is specifically used to perform the following operations:
[0222] At time step T i In the example, the target text features and iterative noise vector are input to the time step T i The corresponding cross-attention sub-network;
[0223] Through the time step T i The corresponding cross-attention sub-network in the noise prediction network performs cross-attention processing on the target text features and the iterative noise vector, and the cross-attention processing results;
[0224] Based on the cross-attention processing results, determine the time step T i The corresponding initial noise vector.
[0225] In a possible implementation, the image prediction module 15 is used to perform cross-attention processing on the target text feature and the iterative noise vector. When the cross-attention processing result is obtained, the image prediction module 15 is specifically used to perform the following operations:
[0226] The initial noise vector is used as the query vector in the cross-attention function, the target text features are used as the key vector in the cross-attention function, and the target text features are used as the value vector in the cross-attention function;
[0227] The product of the query vector and the key vector is determined as the first fused feature through the cross attention function;
[0228] Obtain the number of dimensions corresponding to the fused coding feature, and determine the product of the first fused feature and the inverse of the number of dimensions as the second fused feature;
[0229] Based on the activation sub-function in the cross attention function, the second fused feature is converted into the first activation feature, and the product of the first activation feature and the value vector is determined as the cross attention processing result.
[0230] In a possible implementation, the data processing device 1300 further includes a video determination module 16, and the video determination module 16 is specifically configured to perform the following operations:
[0231] When the video to be generated contains frame images corresponding to P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include a target prediction image;
[0232] A video to be generated, which includes frame images corresponding to P consecutive frame texts and intermediate frame images, is determined as a business video matching the target text.
[0233] In a possible implementation, the frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; the video determination module 16 is used to perform image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images, and the video determination module 16 is specifically used to perform the following operations:
[0234] Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain the optical flow prediction result; frame image P j+1 is the frame image P in the video to be generated j The next frame image, frame image P j+2 is the frame image P in the video to be generated j+1The next frame image, frame image P j-1 is the frame image P in the video to be generated j The previous frame image;
[0235] Based on the frame image P j , frame image P j+1 And the optical flow prediction results, generate the frame image P j and frame image P j+1 The intermediate frame images between.
[0236] For further information, see Fig.14 , Fig.14 A data processing device provided in an embodiment of the present application is shown in FIG. Figure 2 The data processing device 1400 may include: an image sample acquisition module 21 , a noise sample acquisition module 22 , a sample determination module 23 , a sample forward diffusion module 24 , a sample encoding module 25 , a sample reverse diffusion module 26 , and a parameter adjustment module 27 .
[0237] The image sample acquisition module 21 is used to acquire N image samples in the order of video frames, and a text sample respectively associated with each image sample; N is a positive integer;
[0238] The noise sample acquisition module 22 is used to acquire a noise data pool, and determine N noise samples based on the noise data pool and the N image samples; one noise sample is determined by the noise data pool and one image sample;
[0239] A sample determination module 23, used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples;
[0240] The sample forward diffusion module 24 is used to extract features from the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain potential noise features, and to perform forward diffusion processing on the potential noise features to obtain a sample forward noise vector; the target noise sample is determined by the first image sample; the first image sample is the previous image sample of the target image sample in the N image samples;
[0241] The sample encoding module 25 is used to perform text encoding on the target text sample to obtain a first encoding feature, and to perform text encoding on the first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample;
[0242] The sample reverse diffusion module 26 performs reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector;
[0243] The parameter adjustment module 27 is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain the target video generation model; the target video model is used to generate the target video indicated by the target text based on the target text.
[0244] In a possible implementation, the noise data pool includes S noise data, where S is a positive integer; when the noise sample acquisition module 22 is used to determine N noise samples based on the noise data pool and the N image samples, the noise sample acquisition module 22 is specifically used to perform the following operations:
[0245] Obtain a sample sliding step length, obtain N sample noise data groups from the S noise data through the sample sliding step length, and respectively splice the noise data in the N noise data groups to obtain N first sample noise images; the number of noise data included in each sample noise data group is R, and R is a positive integer;
[0246] Based on the N first sample noise maps and the N image samples, N noise samples are generated.
[0247] In a possible implementation, each image sample corresponds to one of N first sample noise maps, and the N image samples include image samples C i , N first sample noise images include image sample C i The corresponding first sample noise map B i , i is a positive integer less than or equal to N; when the noise sample acquisition module 22 is used to generate N noise samples based on the N first sample noise images and the N image samples, the noise sample acquisition module 22 is specifically used to perform the following operations:
[0248] When the image sample C i When the image sample ranked first among N image samples, the first sample noise map B i , determined as image sample C i The corresponding noise sample;
[0249] When the image sample C i If image sample C is not the first image sample in the N image samples, i-1 Perform Fourier transform to obtain image sample C i-1 The first sample frequency domain corresponding to the first sample frequency domain determines the frequency signal less than the sample low-frequency threshold in the first sample frequency domain as the sample low-frequency signal, and the first sample noise map B i Perform Fourier transform to obtain the first sample noise image B iThe corresponding second sample frequency domain, the frequency signal greater than or equal to the sample low-frequency threshold in the second sample frequency domain is determined as the sample high-frequency signal, the sample high-frequency signal is combined with the sample low-frequency signal to obtain a sample full-frequency signal, and the sample full-frequency signal is inverse Fourier transformed to obtain an image sample C i The corresponding noise sample; image sample C i-1 is the image sample C among N image samples i The previous image sample of .
[0250] In a possible implementation, the initial video generation model includes an initial back diffusion network layer; the initial back diffusion network layer is used to perform back diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; the parameter adjustment module 27 is used to adjust the parameters of the initial video generation model according to N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples. When the target video generation model is obtained, the parameter adjustment module 27 is specifically used to perform the following operations:
[0251] According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain the target video generation model, including:
[0252] Generate a model loss value according to N noise samples and predicted noise sample vectors respectively predicted by the N noise samples;
[0253] The parameters of the initial back diffusion network layer are adjusted based on the model loss value to obtain the target video generation model.
[0254] See also Fig.15 , Fig.15 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.15As shown, the computer device 1500 in the embodiment of the present application may include: a processor 1501, a network interface 1504 and a memory 1505. In addition, the above-mentioned computer device 1500 may also include: a user interface 1503, and at least one communication bus 1502. Among them, the communication bus 1502 is used to realize the connection and communication between these components. Among them, the user interface 1503 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1503 may optionally include a standard wired interface and a wireless interface. The network interface 1504 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1505 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1505 may optionally also be at least one storage device located away from the aforementioned processor 1501. As Fig.15 As shown, the memory 1505 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer program.
[0255] The network interface 1504 may provide a network communication element; the user interface 1503 is mainly used to provide an input interface for the user; and the processor 1501 may be used to call a computer program stored in the memory 1505 to execute the steps in the various embodiments of the present application.
[0256] In addition, it should be noted that: the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, the computer program is suitable for being loaded and executed by the processor Figure 3 , Figure 4 or Fig.10 For details on the methods provided in each step, please refer to the Figure 3 , Figure 4 or Fig.10 The implementation methods provided in each step are not repeated here. In addition, the description of the beneficial effects of adopting the same method is not repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the computer program can be deployed to execute on one computer device, or on multiple computer devices located in one place, or on multiple computer devices distributed in multiple locations and interconnected by a communication network.
[0257] The computer-readable storage medium may be the device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0258] The present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes Figure 3 , Figure 4 or Fig.10 The methods provided in the various optional methods are therefore not described in detail here.
[0259] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.
[0260] In the embodiments of the present application, the term "module" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0261] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in this description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0262] The methods and related devices provided by the embodiments of the present application are described with reference to the method flow charts and / or structural diagrams provided by the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer programs. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the computer program executed by the processor of the computer or other programmable device generates a program for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 The computer programs may also be stored in a computer-readable memory that can direct a computer or other programmable device to work in a specific manner, so that the computer program stored in the computer-readable memory produces a product including the program device, or is transmitted through a computer-readable storage medium. The computer program can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The program device is implemented in the process Figure 1 A process or multiple processes and / or structures Figure 1 These computer programs can also be loaded onto a computer or other programmable device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing a computer program executed on the computer or other programmable device for implementing the functions specified in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.
[0263] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.
[0264] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.
[0265] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Obtain a target text, and split the target text to obtain P consecutive frame texts in the video frame sorting order; P is a positive integer; Obtain the target frame text traversed in order from the P continuous frame texts, obtain a noise data pool, and determine a target noise image associated with the target frame text based on a business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is a frame image represented by a business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts; Inputting the target noise image into a target video generation model, performing feature extraction on the target noise image through the target video generation model to obtain target noise features, and performing forward diffusion processing on the target noise features to obtain a target forward noise vector; Inputting the service text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the service text to obtain a second text feature; According to the target forward noise vector, the first text feature and the second text feature, the target noise image is denoised to obtain a target predicted image corresponding to the target frame text, and the target predicted image is used as the next frame image located at the business image in the video to be generated.
2. The method according to claim 1, characterized in that The step of determining a target noise image associated with the target frame text based on the business image and the noise data pool includes: A target splicing noise map associated with the target frame text is obtained through the noise data pool, and a target noise image associated with the target frame text is generated according to a low-frequency signal corresponding to the business image and a high-frequency signal corresponding to the target splicing noise map.
3. The method according to claim 2, characterized in that The noise data pool includes S noise data, where S is a positive integer; the step of obtaining a target splicing noise image associated with the target frame text through the noise data pool, and generating a target noise image associated with the target frame text according to a low-frequency signal corresponding to the business image and a high-frequency signal corresponding to the target splicing noise image, includes: Obtaining a sliding step length, obtaining P noise data groups from the S noise data by using the sliding step length, splicing the noise data in the P noise data groups respectively to obtain P second spliced noise images, and determining a target spliced noise image associated with the target frame text from the P second spliced noise images; the number of noise data included in each noise data group is M, and M is a positive integer; Performing Fourier transform on the service image to obtain a first frequency domain corresponding to the service image, and determining a frequency signal in the first frequency domain that is less than a low-frequency threshold as a low-frequency signal; Performing Fourier transform on the target splicing noise image to obtain a target frequency domain corresponding to the target splicing noise image, and determining a frequency signal in the target frequency domain that is greater than or equal to the low-frequency threshold as a high-frequency signal; The high-frequency signal is combined with the low-frequency signal to obtain a full-frequency signal, and the full-frequency signal is subjected to an inverse Fourier transform to obtain a target noise image associated with the target frame text.
4. The method according to claim 1, characterized in that: The step of performing denoising on the target noise image according to the target forward noise vector, the first text feature and the second text feature to obtain a target predicted image corresponding to the target frame text includes: Performing feature concatenation on the first text feature and the second text feature to obtain a target text feature; In the reverse diffusion network layer in the target video generation model, based on the target text features, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain target prediction noise; T is a positive integer; Determining a target latent vector based on the target prediction noise and the target forward noise vector; The target latent vector is reconstructed by a decoder in the target video generation model to obtain a target predicted image corresponding to the target frame text.
5. The method according to claim 4, characterized in that The step of performing feature concatenation on the first text feature and the second text feature to obtain a target text feature includes: Performing bias processing on the second text feature to obtain an updated text feature with a bias characteristic; the bias characteristic is used to highlight the frame timing relationship between the first text feature and the updated text feature; A target text feature is generated based on the first text feature and the updated text feature.
6. The method according to claim 5, characterized in that The step of generating a target text feature based on the first text feature and the updated text feature includes: Obtaining a context window threshold of the target video generation model, and determining a first text length threshold and a second text length threshold based on the context window threshold; the sum of the first text length threshold and the second text length threshold is equal to the context window threshold; Based on the first text length threshold and the first text feature, a first concatenated text feature is generated; the feature length of the first concatenated text feature is equal to the first text length threshold; Based on the second text length threshold and the updated text feature, a second concatenated text feature is generated; the feature length of the second concatenated text feature is equal to the second text length threshold; The first concatenated text feature and the second concatenated text feature are concatenated to obtain a target text feature.
7. The method according to claim 4, characterized in that The T time steps include time step T i , i is a positive integer less than or equal to T; based on the target text feature, the target forward noise vector is subjected to reverse diffusion processing in T time steps to obtain target prediction noise, including: At the time step T i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector; if the time step T i is the first time step of the T time steps, then the time step T i The corresponding iterative noise vector is the target forward noise vector; The time step T i The corresponding initial noise vector is determined as time step T i+1 The corresponding iterative noise vector at the time step T i+1 In the example, based on the target text features, the time step T i+1 The corresponding iterative noise vector is used for noise prediction to obtain the time step T i+1 The initial noise vector is obtained until the initial noise vector corresponding to the last time step in T time steps is obtained, and the initial noise vector corresponding to the last time step is determined as the target prediction noise; the time step T i+1 is the time step T in the T time steps i The next time step.
8. The method according to claim 7, characterized in that The reverse diffusion network layer includes noise prediction networks corresponding to T time steps, each noise prediction network includes a cross attention sub-network; i In the example, based on the target text features, the time step T i The corresponding iterative noise vector is used for noise prediction to obtain the time step T i The corresponding initial noise vector includes: At the time step T i In the example, the target text feature and the iterative noise vector are input into the time step T i The corresponding cross-attention sub-network; By the time step T i The cross attention sub-network in the corresponding noise prediction network performs cross attention processing on the target text feature and the iterative noise vector, and obtains the cross attention processing result; Based on the cross attention processing result, determine the time step T i The corresponding initial noise vector.
9. The method according to claim 8, characterized in that The cross attention processing is performed on the target text feature and the iterative noise vector, and the cross attention processing result includes: Using the initial noise vector as a query vector in a cross-attention function, using the target text feature as a key vector in the cross-attention function, and using the target text feature as a value vector in the cross-attention function; Determining a product of the query vector and the key vector as a first fused feature through the cross attention function; Obtaining the number of dimensions corresponding to the fused coding feature, and determining the product of the first fused feature and the reciprocal of the number of dimensions as the second fused feature; Based on the activation sub-function in the cross-attention function, the second fused feature is converted into a first activation feature, and the product of the first activation feature and the value vector is determined as the cross-attention processing result.
10. The method according to claim 1, characterized in that Also includes: When the video to be generated includes frame images corresponding to the P consecutive frame texts, image interpolation is performed on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images; the frame images corresponding to the P consecutive frame texts include the target prediction image; The video to be generated, which includes the frame images respectively corresponding to the P consecutive frame texts and the intermediate frame images, is determined as a business video matching the target text.
11. The method according to claim 10, characterized in that The frame images corresponding to the P consecutive frame texts in the video to be generated include frame images P j and frame image P j+1 , j is a positive integer less than P; performing image interpolation on each two adjacent frame images to obtain an intermediate frame image between each two adjacent frame images includes: Based on the frame image P j and frame image P j+1 , determine the frame image P j-1 and frame image P j+2 , for the frame image P j , frame image P j+1 , frame image P j-1 and frame image P j+2 Perform optical flow prediction to obtain an optical flow prediction result; the frame image P j+1 is the frame image P in the video to be generated j The next frame image, the frame image P j+2 is the frame image P in the video to be generated j+1 The next frame image, the frame image P j-1 is the frame image P in the video to be generated j The previous frame image; Based on the frame image P j , the frame image P j+1 and the optical flow prediction result to generate the frame image P j and frame image P j+1 The intermediate frame images between.
12. A data processing method, characterized in that: The method comprises: Obtain N image samples in a video frame sorting order, and a text sample associated with each image sample; N is a positive integer; Acquire a noise data pool, and determine N noise samples based on the noise data pool and the N image samples; one noise sample is determined by the noise data pool and one image sample; Determine a target image sample from the N image samples, and determine a target text sample associated with the target image sample from the N text samples; Performing feature extraction on the target image sample and the target noise sample corresponding to the target image sample among the N noise samples to obtain a potential noise feature, and performing forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; the target noise sample is determined by a first image sample; the first image sample is an image sample previous to the target image sample among the N image samples; Performing text encoding on the target text sample to obtain a first encoding feature, and performing text encoding on the first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample; Performing a reverse diffusion process on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; According to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples, the parameters of the initial video generation model are adjusted to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.
13. The method according to claim 12, characterized in that The noise data pool includes S noise data, where S is a positive integer; and determining N noise samples based on the noise data pool and the N image samples includes: Obtaining a sample sliding step length, obtaining N sample noise data groups from the S noise data by using the sample sliding step length, and splicing the noise data in the N noise data groups respectively to obtain N first sample noise images; the number of noise data included in each sample noise data group is R, and R is a positive integer; Based on the N first sample noise maps and the N image samples, N noise samples are generated.
14. The method according to claim 13, characterized in that Each image sample corresponds to one of the N first sample noise maps, and the N image samples include image samples C i , the N first sample noise images include the image sample C i The corresponding first sample noise map B i , i is a positive integer less than or equal to N; The generating N noise samples based on the N first sample noise images and the N image samples comprises: When the image sample C i When the image sample is ranked first among the N image samples, the first sample noise image B i , determined as the image sample C i The corresponding noise sample; When the image sample C i If the image sample C is not the first image sample in the N image samples, i-1 Perform Fourier transform to obtain the image sample C i-1 The first sample frequency domain corresponding to the first sample frequency domain determines the frequency signal less than the sample low-frequency threshold in the first sample frequency domain as the sample low-frequency signal, and the first sample noise map B i Perform Fourier transform to obtain the first sample noise map B i The corresponding second sample frequency domain is used to determine a frequency signal in the second sample frequency domain that is greater than or equal to the sample low-frequency threshold as a sample high-frequency signal, and the sample high-frequency signal is combined with the sample low-frequency signal to obtain a sample full-frequency signal, and the sample full-frequency signal is subjected to an inverse Fourier transform to obtain the image sample C. i The corresponding noise sample; the image sample C i-1 is the image sample C among the N image samples i The previous image sample of .
15. The method according to claim 12, characterized in that The initial video generation model includes an initial reverse diffusion network layer; the initial reverse diffusion network layer is used to perform reverse diffusion processing on the sample forward noise vector, the first coding feature and the second coding feature to obtain a predicted noise sample vector; The step of adjusting parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors respectively predicted by the N noise samples to obtain a target video generation model comprises: Generate a model loss value according to the N noise samples and predicted noise sample vectors respectively predicted by the N noise samples; The parameters of the initial back diffusion network layer are adjusted based on the model loss value to obtain a target video generation model.
16. A data processing device, characterized in that: The device comprises: The data acquisition module is used to acquire the target text, split the target text, and obtain P consecutive frame texts in the video frame sorting order; P is a positive integer; A data determination module is used to obtain a target frame text traversed in order from the P continuous frame texts, obtain a noise data pool, and determine a target noise image associated with the target frame text based on a business image and the noise data pool; the business image is the last frame image in the video to be generated, and the business image is a frame image represented by a business text; the business text is the previous continuous frame text of the target frame text in the P continuous frame texts; A data forward diffusion module is used to input the target noise image into a target video generation model, perform feature extraction on the target noise image through the target video generation model to obtain target noise features, and perform forward diffusion processing on the target noise features to obtain a target forward noise vector; A feature encoding module, used for inputting the service text and the target frame text into the target video generation model, performing text encoding on the target frame text through the target video generation model to obtain a first text feature, and performing text encoding on the service text to obtain a second text feature; An image prediction module is used to perform feature splicing on the first text feature and the second text feature to obtain a target text feature, perform denoising on the target noise image according to the target forward noise vector and the target text feature to obtain a target predicted image corresponding to the target frame text, and use the target predicted image as the next frame image located at the business image in the video to be generated.
17. A data processing device, characterized in that: The device comprises: The image sample acquisition module is used to acquire N image samples in the order of video frames and text samples respectively associated with each image sample; N is a positive integer; A noise sample acquisition module, used to acquire a noise data pool, and determine N noise samples based on the noise data pool and the N image samples; one noise sample is determined by the noise data pool and one image sample; A sample determination module, used to determine a target image sample from the N image samples, and to determine a target text sample associated with the target image sample from the N text samples; A sample forward diffusion module is used to extract features from the target image sample and the target noise sample corresponding to the target image sample in the N noise samples to obtain a potential noise feature, and to perform forward diffusion processing on the potential noise feature to obtain a sample forward noise vector; the target noise sample is determined by a first image sample; the first image sample is an image sample previous to the target image sample in the N image samples; A sample encoding module, configured to perform text encoding on a target text sample to obtain a first encoding feature, and perform text encoding on a first text sample to obtain a second encoding feature; the first text sample is a text sample associated with the first image sample; The sample reverse diffusion module is used to perform feature splicing on the first coding feature and the second coding feature to obtain the target coding feature, and to perform reverse diffusion processing on the sample forward noise vector and the target coding feature to obtain the predicted noise sample vector A parameter adjustment module is used to adjust the parameters of the initial video generation model according to the N noise samples and the predicted noise sample vectors predicted by the N noise samples to obtain a target video generation model; the target video model is used to generate a target video indicated by the target text based on the target text.
18. A computer device, characterized in that: Includes processor, memory, input and output interfaces; The processor is connected to the memory and the input / output interface respectively, wherein the input / output interface is used to receive and output data, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1-15.
19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 15.
20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.
Citation Information
Cited By
Scene change controllable text video method, system and equipment
CN119583907A
Information processing method and device, computer readable storage medium and electronic equipment
CN119722269A
Video processing method and device, medium and program product
CN119906869A
