Video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video generation method and device, electronic equipment and a storage medium, belongs to the technical field of video generation, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring a sample video and a sample video generation indication text; performing self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise; performing semantic fusion on the target noise based on a preset video generation model, the time step data and the sample video generation indication text to obtain semantic noise; performing data splicing on the semantic noise according to a preset mask to obtain mask noise; training a preset video generation model according to the mask noise and the sample video to obtain a target video generation model; and obtaining a target video generation indication text, and performing video generation on the target video generation indication text according to the target video generation model. According to the embodiment of the invention, the vertically coherent video can be generated based on the text.

Description

Technical Field

[0001] The present application relates to the field of video generation technology, applicable to the field of financial technology, and in particular to a video generation method and device, electronic equipment and storage medium. Background Art

[0002] Generating videos based on text means generating videos based on given text. For example, in the scenario of insurance product marketing, the insurance product introduction text is pre-set, and then the insurance product marketing video is generated based on the text. Usually, generating videos based on text is implemented by a diffusion model. The text input value is fed into the diffusion model, the diffusion model generates video frames, and the video frames are combined to obtain the video. However, since the logic of the diffusion model to generate videos is based on first generating video frames and then synthesizing them, the details and scenes between pictures are not exactly the same, resulting in the generated video being incoherent. Therefore, how to generate a coherent video based on text has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a video generation method and device, an electronic device and a storage medium, aiming to generate a coherent video based on text.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides a video generation method, the method comprising:

[0005] Acquire a sample video and a sample video generation instruction text; wherein the sample video generation instruction text is used to describe the sample video;

[0006] Perform self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise;

[0007] Based on the preset video generation model, the time step data, and the sample video generation instruction text, the target noise is semantically fused to obtain semantic noise;

[0008] Performing data splicing on the semantic noise according to a preset mask to obtain masked noise;

[0009] Training the preset video generation model according to the mask noise and the sample video to obtain a target video generation model;

[0010] Obtain target video generation instruction text, and perform video generation on the target video generation instruction text according to the target video generation model.

[0011] In some embodiments, performing self-attention noise diffusion based on a preset video generation model, preset time step data, and the sample video to obtain target noise includes:

[0012] Generate noise according to the time step data to obtain original noise;

[0013] Performing noise diffusion on the original noise according to the sample video to obtain selected noise;

[0014] Based on the preset video generation model, self-attention calculation is performed on the selected noise to obtain the target noise.

[0015] In some embodiments, performing noise diffusion on the original noise according to the sample video to obtain the selected noise includes:

[0016] Calculating the difference between the frame number of the sample video and the noise number of the original noise to obtain a target diffusion value;

[0017] Diffusion rearranges the original noise according to the target diffusion value to obtain at least one selected noise;

[0018] The original noise is subjected to noise splicing according to all the selected noises to obtain the selected noise.

[0019] In some embodiments, performing self-attention calculation on the selected noise based on the preset video generation model to obtain the target noise includes:

[0020] Performing noise division on the selected noise to obtain at least two sliding window noises;

[0021] Performing self-attention calculation on each sliding window noise based on the preset video generation model to obtain noise feature data of each sliding window noise;

[0022] Calculating the central index value of each sliding window noise to obtain the sliding window central index of each sliding window noise;

[0023] The noise in the sliding window noise is self-attentionally smoothed according to the sliding window center index and the noise feature data of each sliding window noise to obtain the target noise.

[0024] In some embodiments, the sample video generation instruction text includes a scene instruction text and a detail instruction text, and the semantic fusion of the target noise based on the preset video generation model, the time step data, and the sample video generation instruction text to obtain the semantic noise includes:

[0025] If the time step data is between a preset time step upper limit and a preset time step lower limit, performing smooth self-attention fusion on the target noise based on the preset video generation model, the scene indication text and the detail indication text to obtain the semantic noise;

[0026] If the time step data is less than the time step lower limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the scene indication text to obtain the semantic noise;

[0027] If the time step data is greater than the time step upper limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the detail indication text to obtain the semantic noise.

[0028] In some embodiments, the performing smooth self-attention fusion on the target noise based on the preset video generation model, the scene indication text, and the detail indication text to obtain the semantic noise includes:

[0029] The target noise is divided into front-end noise, middle-end noise and back-end noise according to a preset noise index lower limit and a preset noise index upper limit, wherein the index of the front-end noise is less than the preset noise index lower limit, the index of the middle-end noise is between the preset noise index lower limit and the preset noise index upper limit, and the index of the back-end noise is greater than the preset noise index upper limit;

[0030] Based on the preset video generation model and the scene indication text, semantic self-attention fusion is performed on the front noise to obtain the front semantic noise;

[0031] Based on the preset video generation model, the scene indication text and the detail indication text, interpolation semantic self-attention fusion is performed on the mid-segment noise to obtain mid-segment semantic noise;

[0032] Based on the preset video generation model and the detail indication text, semantic self-attention fusion is performed on the rear noise to obtain rear semantic noise;

[0033] The front-end semantic noise, the middle-end semantic noise and the back-end semantic noise are noise-concatenated to obtain the semantic noise.

[0034] In some embodiments, the step of performing data splicing on the semantic noise according to a preset mask to obtain masked noise includes:

[0035] Performing noise masking on the semantic noise according to the preset mask to obtain selected mask noise;

[0036] The preset mask, the semantic noise and the selected mask noise are data spliced ​​to obtain the mask noise.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video generation device, the device comprising:

[0038] A data acquisition module, used to acquire a sample video and a sample video generation instruction text; wherein the video generation instruction text is used to describe the sample video;

[0039] A noise diffusion module, used for performing self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise;

[0040] A semantic fusion module, used to perform semantic fusion on the target noise based on the preset video generation model, the time step data, and the sample video generation instruction text to obtain semantic noise;

[0041] A data splicing module, used for performing data splicing on the semantic noise according to a preset mask to obtain masked noise;

[0042] A model training module, used for training the preset video generation model according to the mask noise and the sample video to obtain a target video generation model;

[0043] The video generation module is used to obtain target video generation instruction text and generate a video for the target video generation instruction text according to the target video generation model.

[0044] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The present application proposes a video generation method and device, an electronic device and a storage medium, which obtains a sample video and a sample video generation instruction text, and then performs self-attention noise diffusion based on a preset video generation model, preset time step data and a sample video to obtain a target noise, and then performs semantic fusion on the target noise based on the preset video generation model, the time step data and the sample video generation instruction text to obtain semantic noise, thereby integrating semantics into the noise, so as to ensure that the video frames generated when the video frames are subsequently generated based on the noise are coherent and consistent and conform to the semantics; further, the semantic noise is data spliced ​​based on a preset mask to obtain masked noise, so that the subsequent video generation model does not completely rely on the context information between adjacent frames when generating video frames based on the noise, but can obtain the context information across frames, thereby avoiding the cascade incoherence problem when generating video frames; finally, the preset video generation model is trained according to the masked noise and the sample video to obtain a target video generation model that can generate a coherent video based on the text, and then the target video generation instruction text is obtained, and the target video generation instruction text is generated according to the target video generation model, so as to obtain a coherent video based on the text. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of a video generation method provided in an embodiment of the present application;

[0048] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0049] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0050] Figure 4 yes Figure 2 Flow chart of step S203 in FIG.

[0051] Figure 5 yes Figure 1 Flow chart of step S103 in FIG.

[0052] Figure 6 yes Figure 5 Flow chart of step S501 in FIG.

[0053] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0054] Figure 8 is a schematic diagram of the structure of a video generating device provided in an embodiment of the present application;

[0055] Fig. 9It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Self-Attention: Self-attention is a deep learning technique that is widely used in neural networks, especially when processing sequence data, such as text analysis or image generation tasks in natural language processing. The self-attention mechanism calculates the relative importance of each element in the input data and assigns a weighted value to each element, thereby determining which information is most critical to the current task. Specifically, the network generates a set of weighted representations that reflect the contribution of each element to the task by interacting with each input element with all other elements. The core idea of ​​self-attention is to "pay attention" to the relationship between a part of the information in the sequence and other parts of the information, rather than just relying on neighboring elements. The self-attention mechanism plays an important role in the Transformer model, greatly improving the efficiency and effectiveness of sequence modeling, especially in tasks such as machine translation, text generation, and sentiment analysis, which can capture long-distance dependencies and process input data in parallel. The advantage of self-attention is that it can significantly improve the performance of the model, especially in the processing of complex sequences or structured data.

[0061] Time Step: In diffusion models, time step refers to the discrete time unit used by the model in the simulation process. Diffusion models usually simulate the gradual change of data through a series of iterative processes, and each time step represents an update stage of the data state in the model. At each time step, the model processes the data according to the previous state, such as by adding and removing noise, and gradually "diffusing" or "restore" the data. Each time step of the diffusion process can bring the data closer to its final target state, usually a clear or recognizable output. The number of time steps and the length of each time step have a direct impact on the performance of the diffusion model. A larger number of time steps can provide more refined step-by-step adjustments, but may also lead to increased computational costs. Time step is a crucial concept in diffusion models. It helps define the speed and accuracy of model iterations and plays a key role in tasks such as generating images and audio.

[0062] U-Net Network: U-Net is a deep learning model. The core feature of U-Net is its unique "U" shaped structure, which consists of an encoder and a decoder. The encoder part gradually extracts the features of the image and reduces the spatial resolution through a series of convolution operations; the decoder part gradually restores the spatial resolution of the image through upsampling and convolution operations. The design of U-Net includes skip connections, which directly pass the low-level features in the encoder to the corresponding layers of the decoder, which can help the model better restore the details of the image. U-Net can maintain high accuracy when processing images with complex details and edges.

[0063] Sliding window: Sliding window is a technique commonly used in signal processing, data stream processing and machine learning. It extracts features or performs calculations by defining a fixed-size window on the data sequence and sliding the window along the data sequence. Each time the window slides, the data in the window changes as the window moves. Usually, it slides one step at a time, or it can slide multiple data points. The sliding window method is widely used in fields such as time series analysis, image processing and text analysis. In image processing, sliding windows are used to identify targets or extract local features by traversing different areas of an image; in natural language processing, sliding windows can be used to generate n-gram features of text to help the model understand the contextual information of the text. The advantage of sliding window technology is that it can efficiently process large-scale data and reduce repeated calculations on the entire data set. It is particularly suitable for online learning and real-time data stream processing.

[0064] Generating videos based on text means generating videos based on given text. For example, in the scenario of insurance product marketing, the insurance product introduction text is pre-set, and then the insurance product marketing video is generated based on the text. Usually, generating videos based on text is implemented by a diffusion model. The text input value is fed into the diffusion model, the diffusion model generates video frames, and the video frames are combined to obtain the video. However, since the logic of the diffusion model to generate videos is based on first generating video frames and then synthesizing them, the details and scenes between pictures are not exactly the same, resulting in the generated video being incoherent. Therefore, how to generate a coherent video based on text has become an urgent problem to be solved.

[0065] Based on this, the embodiments of the present application provide a video generation method and device, an electronic device and a storage medium, which aim to generate a coherent video based on text.

[0066] A video generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video generation method in the embodiments of the present application is described.

[0067] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0068] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0069] The video generation method provided in the embodiment of the present application relates to the field of video generation technology and is applicable to the field of financial technology. The video generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video generation method, etc., but is not limited to the above forms.

[0070] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0071] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0072] Figure 1 is an optional flowchart of the video generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0073] Step S101, obtaining a sample video and a sample video generation instruction text; wherein the sample video generation instruction text is used to describe the sample video;

[0074] Step S102, performing self-attention noise diffusion based on a preset video generation model, preset time step data and sample video to obtain target noise;

[0075] Step S103, semantically fusing the target noise based on the preset video generation model, the time step data, and the sample video generation instruction text to obtain semantic noise;

[0076] Step S104, performing data splicing on the semantic noise according to a preset mask to obtain masked noise;

[0077] Step S105, training a preset video generation model according to the mask noise and the sample video to obtain a target video generation model;

[0078] Step S106, obtaining target video generation instruction text, and performing video generation on the target video generation instruction text according to the target video generation model.

[0079] In the steps S101 to S106 shown in the embodiment of the present application, a sample video and a sample video generation instruction text are obtained, and then self-attention noise diffusion is performed based on a preset video generation model, preset time step data and a sample video to obtain a target noise. Then, semantic fusion is performed on the target noise based on the preset video generation model, the time step data and the sample video generation instruction text to obtain semantic noise, thereby integrating semantics into the noise, so as to ensure that the video frames generated when the video frames are subsequently generated based on the noise are coherent and consistent and conform to the semantics; further, data splicing is performed on the semantic noise based on a preset mask to obtain masked noise, so that the subsequent video generation model does not completely rely on the context information between adjacent frames when generating video frames based on the noise, but can obtain the context information across frames, thereby avoiding the cascade incoherence problem when generating video frames; finally, the preset video generation model is trained according to the masked noise and the sample video to obtain a target video generation model that can generate a coherent video based on the text, and then the target video generation instruction text is obtained, and the target video generation instruction text is generated according to the target video generation model to obtain a coherent video based on the text.

[0080] In step S101 of some embodiments, the sample video is video data, which can be video data generated by a video collector, video production software or manually. The sample video generation instruction text is a descriptive text, which is a descriptive text of the sample video and is used to guide the model to generate videos in the subsequent process. For example, the sample video is a 3-minute skiing video, in which a skier is rapidly sliding down the slope and occasionally performing a rotation. The sample video description text is "a skier rapidly slides down the snowy slope and performs a series of rotations."

[0081] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:

[0082] Step S201, generating noise according to the time step data to obtain original noise;

[0083] Step S202, performing noise diffusion on the original noise according to the sample video to obtain selected noise;

[0084] Step S203, performing self-attention calculation on the selected noise based on a preset video generation model to obtain target noise.

[0085] In steps S201 to S203 shown in the embodiment of the present application, noise is generated through time step data, and then the original noise is diffused according to the sample video, and the content and structure of the noise are gradually adjusted using the self-attention mechanism, thereby achieving fine control of the noise and ensuring that the generated video is not only consistent in each time period, but also maintains coherence between different semantic segments, thereby improving the quality and controllability of subsequently generated videos.

[0086] In step S201 of some embodiments, noise is generated according to the time step data to obtain original noise, and the noise generation is shown in formula (1):

[0087]

[0088] Among them, i is the index of the noise, t is the time step data, x it is the noise value of the i-th noise at the t-th time step data, a t is a preset function value that increases with time steps, x i(t-1) is the noise value of the i-th noise at the t-1th time step, that is, the noise value in the previous cycle, and ∈ is the noise sampled from the standard normal distribution.

[0089] It should be noted that the original noise includes multiple noises, which are [x 1t ,x 2t ,x 3t ……xit ], i is the preset number of initial videos, and the noise amount of the original noise corresponds to the i-frame video.

[0090] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S303:

[0091] Step S301, calculating the difference between the frame number of the sample video and the noise number of the original noise to obtain a target diffusion value;

[0092] Step S302, diffuse and rearrange the original noise according to the target diffusion value to obtain at least one selected noise;

[0093] Step S303: performing noise splicing on the original noise according to all the selected noises to obtain the selected noise.

[0094] In steps S301 to S303 shown in the embodiment of the present application, a target diffusion value is obtained by calculating the difference between the number of frames of the sample video and the noise number of the original noise, and then the original noise is diffused and rearranged according to the target diffusion value to obtain at least one selected noise, and then all the selected noises are noise-concatenated with the original noise to obtain the selected noise, so that the noise number of the selected noise is consistent with the video frame length, and at the same time, the selected noise includes the noise diffused and rearranged from the original noise, so that the selected noise carries more contextual information, providing a data basis for the subsequent generation of a coherent video.

[0095] In steps S301 to S303 of some embodiments, the number of video frames of the sample video is first obtained, then the noise quantity of the original noise is obtained, and then the difference between the noise quantity and the video frame number is calculated to obtain the target diffusion value.

[0096] It should be noted that the noise amount of the original noise is smaller than the number of video frames.

[0097] Specifically, the noise amount of the original noise corresponds to the number of video frames of a video, and the target diffusion value is the difference between the number of video frames corresponding to the original noise and the number of video frames of the sample video.

[0098] Based on the target diffusion value, the original noise is diffused and rearranged to obtain at least one selected noise. For example, if the target diffusion value is 2 and the number of noises of the original noise is 3, then diffusion generation is performed based on the original noise to obtain two preliminary noises. The diffusion rearrangement is shown in formula (2):

[0099] x (i+s)t =shuffle(x (1+s-1)t ,x (2+s-1)t ,x (3+s-1)t ……,x (i+s-1)t) (2),

[0100] Where s is a value that increases by 1 from 1 until it reaches the same value as the target diffusion value. For example, the original noise is (x 1t ,x 2t ,x 3t ), the target diffusion value is 2, x 4t =shuffle(x 1t ,x 2t ,x 3t ),x 5t =shuffle(x 2t ,x 3t ,x 4t ), shuffle() is a shuffle function that is used to randomly arrange the array in an array and then extract the first number in the array. For example, shuffle(x 2t ,x 3t ,x 4t )=(x 3t ,x 4t ,x 2t )→x 3t .

[0101] It should be noted that in the embodiment of the present application, the shuffle extraction range is from (1+s-1) to (i+s-1), and can be set as needed in other embodiments, and the present application does not make any specific restrictions.

[0102] See also Figure 4 In some embodiments, step S203 may include but is not limited to steps S401 to S404:

[0103] Step S401, performing noise division on the selected noise to obtain at least two sliding window noises;

[0104] Step S402, performing self-attention calculation on each sliding window noise based on a preset video generation model to obtain noise feature data of each sliding window noise;

[0105] Step S403, calculating the center index value of each sliding window noise to obtain the sliding window center index of each sliding window noise;

[0106] Step S404, performing self-attention smoothing on the noise in the sliding window noise according to the sliding window center index and noise feature data of each sliding window noise to obtain the target noise.

[0107] In the steps S401 to S404 shown in the embodiment of the present application, by sliding window partitioning the selected noise and performing self-attention calculation, the local correlation between video frames can be effectively captured, and the consistency between the time frames of the generated video can be improved. The self-attention smoothing operation further optimizes the noise characteristics and ensures a smooth transition between different noise areas, thereby improving the quality and detail consistency of the generated video.

[0108] In some embodiments, in step S401 to step S403, the selected noise is divided based on a preset sliding window, and the sliding step length of the sliding window is the same as the window of the sliding window, to obtain at least two sliding window noises. For example, the selected noise is [x 1t ,x 2t ,x 3t ……x 6t ], the sliding window is 2, and the step size is 2, then we get [x 1t ,x 2t ],[x 3t ,x 4t ],[x 5t ,x 6t ]. Then, each sliding window noise is sequentially input into the self-attention layer in the preset video generation model, and the noise feature data of each sliding window is calculated by the self-attention layer of the preset video generation model.

[0109] Then, for each sliding window noise, the index of the middle noise in the sliding window noise is obtained to obtain the sliding window center index of each noise. For example, the sliding window noise is j is the index value of the noise, F j is the jth sliding window noise, i is the index value of the leftmost noise of the sliding window noise, and i+U is the index value of the rightmost noise of the sliding window noise. The sliding window center index is floor() means round down.

[0110] Then, based on the sliding window center index and noise feature data of each sliding window noise, the noise in the sliding window noise is self-attentionally smoothed, as shown in formula (3):

[0111]

[0112] Among them, x i is the self-attention smoothing value of the i-th noise, that is, the index value of the noise in the selected noise, j is the index of the sliding window noise, m j A is the sliding window center index of the sliding window noise corresponding to the noise, that is, the sliding window center index of the jth window, j is the noise characteristic data corresponding to the j-th sliding window noise.

[0113] Each self-attention smoothed value is concatenated to obtain the target noise. For example, the target noise is (x1, x2...x i ).

[0114] See also Figure 5 In some embodiments, the sample video generates instruction text including scene instruction text and detail instruction text, and step S103 includes but is not limited to steps S501 to S503:

[0115] Step S501, if the time step data is between a preset time step upper limit and a preset time step lower limit, smooth self-attention fusion is performed on the target noise based on a preset video generation model, scene indication text and detail indication text to obtain semantic noise;

[0116] Step S502: if the time step data is less than the time step lower limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the scene indication text to obtain semantic noise;

[0117] Step S503: If the time step data is greater than the time step upper limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the detail indication text to obtain semantic noise.

[0118] In steps S501 to S503 shown in the embodiment of the present application, when the time step data is between a preset time step upper limit and a preset time step lower limit, the target noise is smoothed and self-attended based on the preset video generation model, the scene indication text and the detail indication text to obtain semantic noise; when the time step data is less than the time step lower limit, the target noise is semantically self-attended based on the preset video generation model and the scene indication text to obtain semantic noise; when the time step data is greater than the time step upper limit, based on the preset video generation model, the target noise is semantically self-attended based on the preset video generation model and the detail indication text to obtain semantic noise, thereby inserting different semantic information in different time steps of generated noise; in the early stage of the time step, that is, when it is less than the time step lower limit, the target noise is semantically self-attended by the scene indication text; when the time step is between the time step upper limit and the time step lower limit, the scene indication text and the detail indication text are used for smoothing; after the time step upper limit, the detail indication text is used to finalize the details of the noise, so that the video frames finally generated relying on the semantic details are coherent.

[0119] See also Figure 6 In some embodiments, step S501 includes but is not limited to steps S601 to S605:

[0120] Step S601, dividing the target noise into front-end noise, middle-end noise and back-end noise according to a preset noise index lower limit and a preset noise index upper limit, wherein the index of the front-end noise is less than the preset noise index lower limit, the index of the middle-end noise is between the preset noise index lower limit and the preset noise index upper limit, and the index of the back-end noise is greater than the preset noise index upper limit;

[0121] Step S602, performing semantic self-attention fusion on the front noise based on the preset video generation model and the scene indication text to obtain the front semantic noise;

[0122] Step S603, performing interpolation semantic self-attention fusion on the mid-segment noise based on the preset video generation model, the scene indication text, and the detail indication text to obtain the mid-segment semantic noise;

[0123] Step S604, performing semantic self-attention fusion on the rear noise based on the preset video generation model and the detail indication text to obtain the rear semantic noise;

[0124] Step S605 , performing noise splicing on the front-end semantic noise, the middle-end semantic noise and the back-end semantic noise to obtain semantic noise.

[0125] In steps S601 to S605 shown in the embodiment of the present application, the target noise is first divided according to a preset noise index lower limit and a preset noise index upper limit to obtain front noise, middle noise and back noise, and at the same time, semantic self-attention fusion is performed on the front noise based on the scene indication text, and interpolation semantic self-attention fusion is performed on the middle noise based on the scene indication text and the detail indication text to obtain middle semantic noise, and at the same time, semantic self-attention fusion is performed on the back noise based on the detail indication text to obtain back semantic noise, and finally, the front semantic noise, middle semantic noise and back semantic noise are noise-spliced ​​to obtain semantic noise, thereby enhancing the semantic consistency and detail expression in the video generation process, thereby improving the quality and coherence of the generated video.

[0126] In steps S601 to S605 of some embodiments, the target noise is smoothed and self-attentionally fused based on the preset video generation model, the scene indication text, and the detail indication text as shown in formula (4):

[0127]

[0128] Among them, Motion Injection is the semantic attention feature data. In the preset video generation model, it includes the first linear layer Second linear layer The third linear layer And the self-attention layer Attn cross .

[0129] is the embedded text input to the autonomous layer. It changes due to noise. P1 is the scene indication text, and P2 is the detail indication text.

[0130] N γ is the preset noise index lower limit, N τ is the preset upper limit of noise index, and n is the index of noise in the target noise.

[0131] z t is the front segment semantic noise, the middle segment semantic noise or the back segment semantic noise. t When z is the previous semantic noise, z0 is the corresponding previous noise and the corresponding semantic attention feature data. t When z is the middle-segment semantic noise, z0 is the corresponding middle-segment noise and the corresponding semantic attention feature data. t When it is the latter semantic noise, z0 is the corresponding latter noise and the corresponding semantic attention feature data.

[0132] When the target noise is smoothly self-attentively fused based on the preset video generation model, scene indication text and detail indication text, the maximum index of the target noise is less than N. γ The noise of the target noise is obtained by calculating the front noise. At the same time, the middle noise of the target noise that is greater than or equal to the preset noise index lower limit and less than the preset noise index lower limit is obtained. At the same time, the minimum index of the target noise is greater than or equal to N τ The noise of the rear section is obtained.

[0133] For the front noise, use the scene indication text P1 as the embedded text input to the autonomous layer Input to the first linear layer Second linear layer and the third linear layer And the output of the three linear layers is input into the self-attention layer Attn cross In the layer, the semantic attention feature data Motion Injection corresponding to the previous noise is obtained.

[0134] For the mid-segment noise, the scene indication text P1 and detail indication text P2 are interpolated and smoothed to obtain the embedded text input to the main layer. Input to the first linear layer Second linear layer and the third linear layer And the output of the three linear layers is input into the self-attention layer Attn cross In the layer, the semantic attention feature data Motion Injection corresponding to the middle noise is obtained.

[0135] For the latter noise, use the detail indication text P2 as the embedded text input to the main layer Input to the first linear layer Second linear layer and the third linear layer And the output of the three linear layers is input into the self-attention layer Attn cross In the layer, the semantic attention feature data Motion Injection corresponding to the subsequent noise is obtained.

[0136] The front-segment semantic noise is obtained by multiplying the front-segment noise and the corresponding semantic attention feature data Motion Injection.

[0137] The mid-segment semantic noise is obtained by multiplying the mid-segment noise and the corresponding semantic attention feature data Motion Injection.

[0138] The latter-segment semantic noise is obtained by multiplying the latter-segment noise and the corresponding semantic attention feature data Motion Injection.

[0139] Then the front semantic noise, the middle semantic noise and the back semantic noise are concatenated to obtain semantic noise.

[0140] In step S502 of some embodiments, when the time step data is less than the time step lower limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the scene indication text. The semantic self-attention fusion is shown in formula (5):

[0141]

[0142] Among them, Motion Injection is the semantic attention feature data. In the preset video generation model, it includes the first linear layer Second linear layer The third linear layer And the self-attention layer Attn cross .

[0143] P1 is the scene indication text. t is semantic noise, and z0 is target noise.

[0144] In step S503 of some embodiments, the time step data is greater than the time step upper limit, and semantic self-attention fusion is performed on the target noise based on the preset video generation model and the detail indication text to obtain semantic noise. The semantic self-attention fusion is shown in formula (6):

[0145]

[0146] Among them, Motion Injection is the semantic attention feature data. In the preset video generation model, it includes the first linear layer Second linear layer The third linear layer And the self-attention layer Attn cross .

[0147] P2 is the scene indication text. t is semantic noise, and z0 is target noise.

[0148] See also Figure 7 In some embodiments, step S104 may include but is not limited to steps S701 to S702:

[0149] Step S701, performing noise masking on semantic noise according to a preset mask to obtain selected mask noise;

[0150] Step S702 , concatenating the preset mask, the semantic noise and the selected mask noise to obtain the mask noise.

[0151] In steps S701 to S702 shown in the embodiment of the present application, the semantic noise is masked according to a preset mask to obtain a selected mask noise, and then the preset mask, the semantic noise and the selected mask noise are data spliced ​​to obtain the mask noise, so that the subsequent video generation model does not completely rely on the context information between adjacent frames when generating video frames based on noise, but can obtain the context information across frames, thereby avoiding the cascade incoherence problem when generating video frames.

[0152] In step S701 of some embodiments, the noise mask is obtained by multiplying the semantic noise according to a preset mask to obtain a selected mask noise.

[0153] In step S702 of some embodiments, data splicing is to splice the preset mask, semantic noise and selected mask noise together, as shown in formula (7):

[0154]

[0155] Among them, z t is semantic noise, m is the preset mask, is the selected mask noise, is the mask noise.

[0156] In step S105 of some embodiments, the mask noise is input into a preset video generation model for mask decoding to obtain a predicted video, and then a loss value is calculated based on the video frames of the predicted video and the video frames of the sample video to obtain a training loss value, and then the parameters of the preset video generation model are optimized according to the training loss value to obtain a target video generation model.

[0157] In one embodiment, the preset video generation model includes a first self-attention layer, a second self-attention layer and a Unet network connected in sequence, wherein the first self-attention layer is used to perform self-attention noise diffusion according to time step data and sample videos, the second self-attention layer is used to generate instruction text according to the time step data and sample videos to semantically fuse the target noise, and the Unet network is used to perform mask decoding according to mask noise.

[0158] In step S106 of some embodiments, target video generation instruction text is obtained, and then the target video generation instruction text is input into a target video generation model to perform video generation to obtain a target video.

[0159] See also Figure 8 The embodiment of the present application further provides a video generating device, which can implement the above-mentioned video generating method, and the device includes:

[0160] The data acquisition module 801 is used to acquire a sample video and a sample video generation instruction text; wherein the video generation instruction text is used to describe the sample video;

[0161] A noise diffusion module 802 is used to perform self-attention noise diffusion based on a preset video generation model, preset time step data and sample video to obtain target noise;

[0162] The semantic fusion module 803 is used to perform semantic fusion on the target noise based on the preset video generation model, time step data, and sample video generation instruction text to obtain semantic noise;

[0163] The data splicing module 804 is used to perform data splicing on the semantic noise according to the preset mask to obtain mask noise;

[0164] A model training module 805 is used to train a preset video generation model according to the mask noise and the sample video to obtain a target video generation model;

[0165] The video generation module 806 is used to obtain the target video generation instruction text and perform video generation on the target video generation instruction text according to the target video generation model.

[0166] The specific implementation of the video generating device is basically the same as the specific implementation of the above-mentioned video generating method, and will not be repeated here.

[0167] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above video generation method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0168] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0169] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0170] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video generation method of the embodiment of this application;

[0171] Input / output interface 903, used to implement information input and output;

[0172] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0173] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0174] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0175] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned video generation method is implemented.

[0176] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0177] The video generation method, video generation device, electronic device and storage medium provided by the embodiments of the present application obtain sample videos and sample video generation instruction text, then perform self-attention noise diffusion based on a preset video generation model, preset time step data and sample video to obtain target noise, then perform semantic fusion on the target noise based on the preset video generation model, time step data and sample video generation instruction text to obtain semantic noise, thereby integrating semantics into the noise, so as to ensure that the video frames generated when the video frames are subsequently generated based on the noise are coherent and consistent with the semantics; further, data splicing is performed on the semantic noise based on a preset mask to obtain masked noise, so that the subsequent video generation model does not completely rely on the context information between adjacent frames when generating video frames based on the noise, but can obtain context information across frames, thereby avoiding cascading incoherence problems when generating video frames; finally, the preset video generation model is trained according to the masked noise and the sample video to obtain a target video generation model that can generate a coherent video based on the text, then the target video generation instruction text is obtained, and video generation is performed on the target video generation instruction text according to the target video generation model, so as to obtain a coherent video based on the text.

[0178] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0179] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0182] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0183] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0188] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video generation method, characterized in that: The method comprises: Acquire a sample video and a sample video generation instruction text; wherein the sample video generation instruction text is used to describe the sample video; Perform self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise; Based on the preset video generation model, the time step data, and the sample video generation instruction text, the target noise is semantically fused to obtain semantic noise; Performing data splicing on the semantic noise according to a preset mask to obtain masked noise; Training the preset video generation model according to the mask noise and the sample video to obtain a target video generation model; Obtain target video generation instruction text, and perform video generation on the target video generation instruction text according to the target video generation model.

2. The method according to claim 1, characterized in that The method of performing self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise includes: Generate noise according to the time step data to obtain original noise; Performing noise diffusion on the original noise according to the sample video to obtain selected noise; Based on the preset video generation model, self-attention calculation is performed on the selected noise to obtain the target noise.

3. The method according to claim 2, characterized in that The step of performing noise diffusion on the original noise according to the sample video to obtain the selected noise includes: Calculating the difference between the frame number of the sample video and the noise number of the original noise to obtain a target diffusion value; Diffusion rearranges the original noise according to the target diffusion value to obtain at least one selected noise; The original noise is subjected to noise splicing according to all the selected noises to obtain the selected noise.

4. The method according to claim 2, characterized in that: The performing self-attention calculation on the selected noise based on the preset video generation model to obtain the target noise includes: Performing noise division on the selected noise to obtain at least two sliding window noises; Performing self-attention calculation on each sliding window noise based on the preset video generation model to obtain noise feature data of each sliding window noise; Calculating the central index value of each sliding window noise to obtain the sliding window central index of each sliding window noise; The noise in the sliding window noise is self-attentionally smoothed according to the sliding window center index and the noise feature data of each sliding window noise to obtain the target noise.

5. The method according to claim 1, characterized in that The sample video generation instruction text includes a scene instruction text and a detail instruction text, and the target noise is semantically fused based on the preset video generation model, the time step data, and the sample video generation instruction text to obtain semantic noise, including: If the time step data is between a preset time step upper limit and a preset time step lower limit, performing smooth self-attention fusion on the target noise based on the preset video generation model, the scene indication text and the detail indication text to obtain the semantic noise; If the time step data is less than the time step lower limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the scene indication text to obtain the semantic noise; If the time step data is greater than the time step upper limit, semantic self-attention fusion is performed on the target noise based on the preset video generation model and the detail indication text to obtain the semantic noise.

6. The method according to claim 5, characterized in that The step of performing smooth self-attention fusion on the target noise based on the preset video generation model, the scene indication text, and the detail indication text to obtain the semantic noise includes: The target noise is divided into front-end noise, middle-end noise and back-end noise according to a preset noise index lower limit and a preset noise index upper limit, wherein the index of the front-end noise is less than the preset noise index lower limit, the index of the middle-end noise is between the preset noise index lower limit and the preset noise index upper limit, and the index of the back-end noise is greater than the preset noise index upper limit; Based on the preset video generation model and the scene indication text, semantic self-attention fusion is performed on the front noise to obtain the front semantic noise; Based on the preset video generation model, the scene indication text and the detail indication text, interpolation semantic self-attention fusion is performed on the mid-segment noise to obtain mid-segment semantic noise; Based on the preset video generation model and the detail indication text, semantic self-attention fusion is performed on the rear noise to obtain rear semantic noise; The front-end semantic noise, the middle-end semantic noise and the back-end semantic noise are noise-concatenated to obtain the semantic noise.

7. The method according to any one of claims 1 to 4, characterized in that: The step of performing data splicing on the semantic noise according to the preset mask to obtain masked noise includes: Performing noise masking on the semantic noise according to the preset mask to obtain selected mask noise; The preset mask, the semantic noise and the selected mask noise are data spliced ​​to obtain the mask noise.

8. A video generating device, characterized in that: The device comprises: A data acquisition module, used to acquire a sample video and a sample video generation instruction text; wherein the video generation instruction text is used to describe the sample video; A noise diffusion module, used for performing self-attention noise diffusion based on a preset video generation model, preset time step data and the sample video to obtain target noise; A semantic fusion module, used to perform semantic fusion on the target noise based on the preset video generation model, the time step data, and the sample video generation instruction text to obtain semantic noise; A data splicing module, used for performing data splicing on the semantic noise according to a preset mask to obtain masked noise; A model training module, used for training the preset video generation model according to the mask noise and the sample video to obtain a target video generation model; The video generation module is used to obtain target video generation instruction text and generate a video for the target video generation instruction text according to the target video generation model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the video generation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video generating method according to any one of claims 1 to 7 is implemented.

Images