Video generation method, computing device, computer-readable storage medium, and computer program product

Abstract

Embodiments of the present specification provide a video generation method, a computing device, a computer readable storage medium and a computer program product, wherein the video generation method comprises: receiving a video generation instruction, and generating an initial prompt text according to the video generation instruction; obtaining a reference noise map set, inputting the reference noise map set and the initial prompt text into a first video generation model, obtaining a reference video generated by the first video generation model, and obtaining a target video feature information set generated by the first video generation model in the process of generating the reference video; receiving a video adjustment instruction, and generating a target prompt text according to the video adjustment instruction and the initial prompt text; inputting the reference noise map set, the target prompt text and the target video feature information set into a second video generation model, and obtaining a target video generated by the second video generation model. The target video is generated by the target video feature information set generated in the process of generating the reference video, which improves the accuracy of video editing.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a video generation method. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, text-based video editing has become one of the most eye-catching and widely used technologies in this field. This technology edits videos by using text as input, aiming to enhance the visual appeal and creativity of video content. In particular, with the improvement of text-to-image generation models, text-to-video generation technology has also been significantly improved.

[0003] Currently, when users use slightly modified text to regenerate a video based on the original video, the resulting video often differs significantly from the original, limiting content modification based on the original video. Therefore, to address this issue, a video generation method is needed that allows for precise editing of the original video using text commands. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a video generation method, a video generation method applied to a cloud server. One or more embodiments of this specification also relate to a video generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a video generation method is provided, comprising:

[0006] Receive video generation instructions and generate initial prompt text based on the video generation instructions;

[0007] Obtain a reference noise map group, input the reference noise map group and the initial prompt text into the first video generation model, obtain the reference video generated by the first video generation model, and obtain the target video feature information set generated by the first video generation model in the process of generating the reference video;

[0008] Receive video adjustment instructions, and generate target prompt text based on the video adjustment instructions and the initial prompt text;

[0009] The reference noise map group, the target prompt text, and the target video feature information set are input into the second video generation model to obtain the target video generated by the second video generation model.

[0010] According to a second aspect of the embodiments of this specification, a video generation method applied to a cloud server is provided, comprising:

[0011] The receiving end device sends a video generation command, and generates an initial prompt text based on the video generation command;

[0012] Obtain a reference noise map group, input the reference noise map group and the initial prompt text into the first video generation model, obtain the reference video generated by the first video generation model, and obtain the target video feature information set generated by the first video generation model in the process of generating the reference video;

[0013] The receiver sends a video adjustment command from the receiving end device, and generates a target prompt text based on the video adjustment command and the initial prompt text;

[0014] The reference noise map set, the target prompt text, and the target video feature information set are input into a second video generation model to obtain the target video generated by the second video generation model, and the target video is returned to the edge device. According to a third aspect of an embodiment of this specification, a computing device is provided, comprising:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the above-described video generation method and the video generation method applied to the cloud server.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the video generation method described above and the video generation method applied to a cloud server.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the video generation method described above and the video generation method applied to a cloud server.

[0019] One embodiment of this specification implements receiving a video generation instruction and generating initial prompt text according to the video generation instruction; obtaining a reference noise map group, inputting the reference noise map group and the initial prompt text into a first video generation model to obtain a reference video generated by the first video generation model, and obtaining a target video feature information set generated by the first video generation model in the process of generating the reference video; receiving a video adjustment instruction and generating target prompt text according to the video adjustment instruction and the initial prompt text; inputting the reference noise map group, the target prompt text, and the target video feature information set into a second video generation model to obtain a target video generated by the second video generation model.

[0020] By applying the solution of the embodiments in this specification, a reference video is generated from the initial prompt text, and the set of target video feature information generated during the generation of the reference video is used to generate the target video based on the target prompt text. This ensures that the target video generated from the target prompt text, which is slightly modified from the initial prompt text, maintains a small difference from the reference video, thereby improving the accuracy of video editing. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a video generation method provided in one embodiment of this specification;

[0022] Figure 2 This is a schematic diagram of a process for generating video editing, provided by one embodiment of this specification;

[0023] Figure 3 This is a schematic diagram of a process for editing original video, provided in one embodiment of this specification;

[0024] Figure 4 This is a schematic diagram of a workflow for video editing with an additional mask, provided in one embodiment of this specification.

[0025] Figure 5 This is a flowchart illustrating a video generation method applied to a cloud server, as provided in one embodiment of this specification.

[0026] Figure 6 This is an architecture diagram of a video generation system provided in one embodiment of this specification;

[0027] Figure 7 This is a flowchart illustrating a process for editing an existing user video, provided in one embodiment of this specification.

[0028] Figure 8 This is a schematic diagram of the structure of a video generation device provided in one embodiment of this specification;

[0029] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0030] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0031] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0033] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0035] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0036] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0037] Deterministic Denoising Diffusion Implicit Models (DDIM) are generative models based on diffusion models, characterized by a deterministic approach to sample generation. Unlike traditional random diffusion models, DDIM's output at each step of the denoising process is deterministic, not random. This method not only speeds up the sampling process but also allows for more accurate control over the quality of generated samples. DDIM gradually recovers clear data from noisy data through a series of inverse diffusion steps and is widely used in image generation, super-resolution, and other fields.

[0038] Stable Diffusion is an advanced deep learning model specifically designed for generating high-quality images. It combines diffusion models with deep learning techniques to generate images by progressively removing noise, thus enabling the generation of complex and high-resolution images. A key advantage of Stable Diffusion is its ability to stably generate images with rich detail and realism while maintaining high computational efficiency. This has led to its widespread application in fields such as image compositing, artistic creation, and media editing.

[0039] A Variational Autoencoder (VAE) is a deep learning model used for unsupervised learning tasks, particularly data generation. It consists of two parts: an encoder and a decoder. The encoder transforms the input data into a representation in the latent space, and the decoder reconstructs the data from this latent representation. A VAE is trained by minimizing the reconstruction error and the difference between the latent representation and the prior distribution (typically a Gaussian distribution). This allows the VAE to both efficiently generate data and learn the complex distribution of the data. VAEs are widely used in image generation, denoising, style transfer, and other fields.

[0040] This specification provides a video generation method, a video generation method applied to a cloud server, and also relates to a video generation apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0041] See Figure 1 , Figure 1 A flowchart of a video generation method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0042] Step 102: Receive video generation instructions and generate initial prompt text according to the video generation instructions.

[0043] In practical applications, the video generation command is used to generate a reference video, and the initial prompt text is a description of the reference video. Specifically, the video generation command can be understood as an instruction to generate a reference video that the user wants to modify. For example, after a user uploads the original video (Video 1) on the video editing service page and clicks "Modify with this video," the command sent to the server is a video generation command. Alternatively, it could be the command sent to the server after the user enters "Generate a video focusing on rabbits" and clicks send, etc. This manual does not impose any restrictions on this.

[0044] The initial prompt text can be understood as standardized text used to describe the target video that the user wants to modify. For example, based on the original video uploaded by the user: Video 1, a description of the video is generated as "a video of a rabbit eating a watermelon"; another example is to parse the text entered by the user as "generate a video that focuses on rabbits" and generate a standardized description as "a photo video of rabbits", etc. This manual does not impose any restrictions on this.

[0045] By receiving video generation instructions, a standardized initial prompt text can be generated based on those instructions. This makes the video generated by the video generation model more accurate. Furthermore, the standardized initial prompt text allows subsequent adjustments to the video to be made using another standardized text, thereby improving the accuracy of video editing.

[0046] Considering that users will first generate a video in the form of text, and then modify the video based on the initial text, the video generation instruction carries the initial text information;

[0047] Furthermore, the system receives a video generation instruction and generates initial prompt text based on the video generation instruction, including:

[0048] Parse the video generation instructions to obtain the initial text information;

[0049] The initial prompt text is generated based on the initial text information;

[0050] Accordingly, a reference noise map set is obtained, including:

[0051] The reference noise map group is generated by using preset noise map group generation rules;

[0052] Accordingly, receive video adjustment instructions, including:

[0053] Receive the video adjustment instruction corresponding to the reference video.

[0054] In practical applications, the initial text information is the text information directly input by the user, the reference noise map group is a set of noise maps used to generate the reference video, the noise map group generation rule is the rule for generating the noise map group when the video generation command sent by the user carries the initial text information, the reference video is the video generated based on the reference noise map group and the initial prompt text, and the video adjustment command is the command to adjust the reference video.

[0055] Specifically, the initial text can be understood as non-standard information directly input by the user, such as "Generate a video focusing on a rabbit" or "Give me a video of a rabbit jumping." When the video generation instruction carries initial text information, the obtained initial prompt text can be understood as standardized text information that highlights the key points of the initial text sent by the user. For example, if the initial text sent by the user is "Generate a video focusing on a rabbit," then the initial prompt text determined after parsing the initial text is "A video of rabbit photos." This initial prompt text highlights the key point about the rabbit in the initial text and interprets "focusing on the rabbit" as "rabbit photos," making the generated video more accurate.

[0056] When the video generation instruction carries initial text information, the obtained reference noise map set can be understood as a set of random noise images, such as a set of Gaussian noise maps, a set of uniform noise maps, a set of speckled noise maps, etc. This specification does not impose any restrictions on this. The noise map set generation rule can be understood as the rule for generating a set of random noise maps, such as the Gaussian noise map generation rule, the uniform noise map generation rule, etc. The reference video can be understood as a video generated based on the initial prompt text, which conforms to the description of the initial prompt text. For example, if the initial prompt text is "a video of a rabbit's photos," the generated reference video is Video A, which is a video of a rabbit resting on the grass, and there is only a rabbit on the grass.

[0057] The video generation command sent by the user carries initial text information. This can be understood as the user needing to first generate a video using the initial text information, and then edit the generated video. (Reference) Figure 2 , Figure 2 This specification illustrates a flowchart of a video editing process according to an embodiment of the present invention. The process involves receiving initial text information from a user and parsing it into standardized initial prompt text 204. A set of Gaussian noise reference noise maps 202 is then generated using preset rules. The initial prompt text 204 and the reference noise maps 202 are then input into the cross-feature extraction unit of the first feature extraction layer 2062 of the first video generation model 206 to obtain cross-feature information that integrates the noise map features and the initial prompt text features. This cross-feature information is then input into the self-attention extraction unit for self-attention extraction to obtain self-attention features that enhance key positions in the noise map. The cross-feature information is then input into the spatiotemporal self-attention extraction unit to obtain spatiotemporal feature information containing the spatiotemporal features of the reference noise maps. Finally, the cross-feature information, self-attention features, and spatiotemporal feature information are input into the first video generation layer 2064 to obtain a reference video 208.

[0058] After receiving the reference video 208, the user issues a modification command to the reference video 208, sending a video adjustment instruction. Then, based on the adjustment text in the video adjustment instruction, a target prompt text 216 with slight variations from the initial prompt text 204 is generated. Subsequently, the same reference noise map group 202 and target prompt text 216 are input into the second video generation model 218. Since the second video generation model 218 has the same structure and parameters as the first video generation model 206, the operations of the reference noise map group 202 and the target prompt text 216 in the second feature extraction layer 2182 and the operations of the reference noise map group 202 and the initial prompt text 204 in the first feature extraction layer 2062 are identical. Then, the problem feature information and key value feature information in the self-attention features generated based on the target prompt text 216 and the reference noise map group 202 are replaced with reference problem feature information 210 and reference key value feature information 212 generated based on the initial prompt text 204 and the reference noise map group 202. Then, the spatiotemporal feature information in the spatiotemporal feature information generated based on the target prompt text 216 and the reference noise map group 202 is replaced with reference spatiotemporal feature information 214 generated based on the initial prompt text 204 and the reference noise map group 202. The replaced feature information is input into the second video generation layer 2184 to obtain the target video 220 obtained by the user based on the reference video 208.

[0059] By receiving a video generation instruction carrying initial text information, the system first generates a video in text form, and then modifies the video based on the initial text, thereby enabling users to edit the video.

[0060] Considering that users will directly upload an original video and then modify it, the video generation instruction carries the initial video information.

[0061] Furthermore, the system receives a video generation instruction and generates initial prompt text based on the video generation instruction, including:

[0062] Parse the video generation instruction to obtain the initial video information;

[0063] The initial prompt text is generated based on the initial video information, wherein the initial prompt text is used to describe the initial video information;

[0064] Accordingly, a reference noise map set is obtained, including:

[0065] Based on the initial video information, the reference noise map group corresponding to the initial video information is generated;

[0066] Accordingly, receive video adjustment instructions, including:

[0067] Receive the video adjustment instruction corresponding to the initial video information.

[0068] In practical applications, the initial video information is the video uploaded by the user. Specifically, the initial video information can be understood as the video that the user wants to edit.

[0069] When the video generation instruction carries initial video information, the obtained reference noise map set can be understood as a set of noise maps generated by noise processing each frame in the initial video information. Specifically, the noise processing method for the frames in the initial video information can be reverse DDIM (Deterministic Denoising Diffusion Implicit Models), etc. This specification does not impose any restrictions on this.

[0070] When the video generation instruction carries initial video information, the obtained initial prompt text can be understood as a summary of the initial video information, obtaining a text describing the initial video. For example, if the video uploaded by the user is a video of a rabbit eating watermelon on a table, then the initial prompt text corresponding to the video is "a video of a rabbit eating watermelon on a table".

[0071] By using a set of noise maps obtained from the initial video information and the descriptive text of that initial video information, a reference video generated based on the above data can be made almost identical to the initial video. Therefore, the feature values ​​generated when generating the reference video can also be understood as the feature values ​​of the initial video information. Subsequently, the target video generated using these feature values ​​will not differ much from the initial video.

[0072] The video generation command sent by the user carries initial video information, which can be understood as the user needing to edit the original video. However, since the original video has not been generated by the first video generation model, a reference video that is essentially the same as the original video needs to be generated first using the first video generation model to obtain some features from the original video. Figure 3 , Figure 3This document illustrates a flowchart of an embodiment of a process for editing original video. The process involves receiving initial video information 302 from a user, then processing noise in each frame of the initial video information 302 to generate a set of reference noise maps 304. The initial video information 302 is then parsed into initial prompt text 306 describing the initial video information. The initial prompt text 306 and the reference noise maps 304 are then input into the cross-feature extraction unit of the first feature extraction layer 3082 in the first video generation model 308 to obtain cross-feature information that integrates noise map features and initial prompt text features. This cross-feature information is then input into the self-attention extraction unit for self-attention extraction to obtain self-attention features that enhance key positions in the noise maps. The cross-feature information is then input into the spatiotemporal self-attention extraction unit to obtain spatiotemporal feature information containing the spatiotemporal features of the reference noise maps. Finally, the cross-feature information, self-attention features, and spatiotemporal feature information are input into the first video generation layer 3084 to obtain a reference video 310.

[0073] The user gives a command to modify the initial video information 302, sending a video adjustment instruction. Then, based on the adjustment text in the video adjustment instruction, a target prompt text 318 with slight modifications to the initial prompt text 306 is generated. Next, the same reference noise map group 304 and target prompt text 318 are input into the second video generation model 320. Since the second video generation model 320 has the same structure and parameters as the first video generation model 308, the reference noise map group 304 and target prompt text 318 are processed in the second feature extraction layer 3202, and the operations of the reference noise map group 304 and initial prompt text 306 in the first feature extraction layer 3082 are the same. Then, based on... The problem feature information and key value feature information in the self-attention features generated by the target prompt text 318 and the reference noise map group 304 are replaced with reference problem feature information 312 and reference key value feature information 314 generated by the initial prompt text 306 and the reference noise map group 304. Then, the spatiotemporal feature information in the spatiotemporal feature information generated by the target prompt text 318 and the reference noise map group 304 is replaced with reference spatiotemporal feature information 316 generated by the initial prompt text 306 and the reference noise map group 304. The replaced feature information is input into the second video generation layer 3204 to obtain the target video 322 obtained by the user by modifying the reference video 310.

[0074] By receiving video generation instructions carrying initial video information, users can modify existing videos, thus enabling them to edit those videos.

[0075] Step 104: Obtain a reference noise map group, input the reference noise map group and the initial prompt text into the first video generation model, and obtain the target video feature information set generated by the first video generation model in the process of processing the reference noise map group and the initial prompt text.

[0076] In practical applications, the first video is a model used to extract features of the part of the video to be modified, and the target video feature information set is the part of the features of the video to be modified.

[0077] Specifically, the initial prompt text describing the video to be modified is input into the first video generation model. The first video generation model can process the initial prompt text to extract some features of the video to be modified. By using the extracted features to generate the target video, the generated target video can be made more similar to the unmodified parts of the video to be modified, thereby improving the accuracy of video modification.

[0078] Furthermore, the reference noise map group and the initial prompt text are input into the first video generation model to obtain a set of target video feature information generated by the first video generation model in the process of processing the reference noise map group and the initial prompt text, including:

[0079] The reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video;

[0080] Obtain the set of target video feature information generated by the first video generation model during the process of generating the reference video.

[0081] In practical applications, the reference video is the video that needs to be modified.

[0082] In other words, the first video generation model can be further understood as a model used to extract features from a reference video, which can be understood as the video to be modified. Inputting the initial prompt text describing the video to be modified into the first video generation model to generate the reference video can be understood as extracting some features from the reference video during the generation process. By using these extracted features to generate the target video, the generated target video can be made more similar to the unmodified parts of the reference video, thereby improving the accuracy of video modification.

[0083] The target video feature information set can be understood as a subset of the features of the reference video. The target video feature set may include one or more features of the reference video, such as temporal attention features representing the temporal features of each entity in the reference video, reference spatial attention features representing the spatial features of each entity in the reference video, reference spatiotemporal attention features representing the temporal and spatial fusion features of each entity in the reference video, reference self-attention features representing the relationship between each pixel in the reference video, reference cross-attention features representing the association between the reference video and the initial prompt text, etc. This specification does not impose any limitations on this.

[0084] In one embodiment provided in this specification, the initial prompt text "a rabbit photo video" and a set of Gaussian noise maps are input into the first video generation model. The obtained reference video is a video of a white rabbit resting on the grass. In the process of generating the video, the spatiotemporal attention features representing the temporal and spatial fusion features of each entity in the video and the self-attention features representing the relationship between each pixel in the video are obtained.

[0085] By extracting features corresponding to the reference video generated during the reference video generation process, and then using the obtained features to generate the target video, the generated target video can be made more similar to the unmodified parts of the aforementioned reference video.

[0086] Considering that generating a video requires automatically discovering and extracting useful information or features from prompt text and noisy images for further processing and analysis, the first video generation model includes a first feature extraction layer and a first video generation layer.

[0087] Furthermore, the reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video, including:

[0088] The reference noise map group and the initial prompt text are input into the first feature extraction layer to obtain the reference attention feature information set output by the first feature extraction layer;

[0089] The reference attention feature information set is input into the first video generation layer to obtain the reference video output by the first video generation layer.

[0090] In practical applications, the first feature extraction layer is used to extract the features of the reference video, the first video generation layer is used to generate a deep learning network layer for the reference video based on the features extracted by the first feature extraction layer, and the reference attention feature information set is used as input to the video generation layer to obtain the reference video.

[0091] Specifically, the first feature extraction layer can be understood as the computational layer that generates the features of the reference video. This layer can be a feature extraction layer with a neural network structure or a feature extraction layer without a neural network structure. This specification does not impose any restrictions on the results of this layer. For example, based on the input initial prompt text and a set of noise maps, the self-attention features of the relationship between each pixel in the video to be generated, the cross-attention features of the relationship between the video to be generated and the initial prompt text, etc., are extracted. This specification does not impose any restrictions on this.

[0092] The first video generation layer can be understood as a deep learning network layer that generates the corresponding video based on the extracted video features. For example, network models that can generate videos include DDIM, Static Diffusion, etc. This specification does not impose any restrictions on this.

[0093] The reference attention feature information set can be understood as all the features of the reference video to be generated. It includes all the features about the reference video extracted by the first feature extraction layer, such as temporal attention features, spatial attention features, spatiotemporal attention features, self-attention features, cross-attention features, etc. This specification does not impose any restrictions on this.

[0094] The first feature extraction layer extracts all features from the reference video. Features that the user will not modify can be obtained from the extracted features and used to generate the target video. This makes the generated target video more similar to the unmodified parts of the reference video.

[0095] Furthermore, the set of target video feature information generated by the first video generation model during the generation of the reference video is obtained, including:

[0096] Obtain a set of reference attention feature information output by the first feature extraction layer, wherein the set of reference attention feature information includes at least one set of attention feature information;

[0097] A target video feature information set is determined based on the reference attention feature information set, wherein the target video feature information set includes at least one attention feature information.

[0098] Specifically, the target video feature information set can be understood as the selected features that the user will not modify from the reference attention feature information set representing all features of the reference video. By obtaining these user-unmodified features and using them to generate the target video, the generated target video can be made more similar to the unmodified parts of the aforementioned reference video.

[0099] Preferably, considering that users usually do not modify the spatiotemporal relationships between entities in the reference video, or the relationships between elements in the reference video, the reference attention feature information set includes reference self-attention feature information and reference spatiotemporal self-attention feature information.

[0100] Furthermore, determining the target video feature information set based on the reference attention feature information set includes:

[0101] Based on the aforementioned set of reference attention feature information, determine the reference self-attention layer feature information and the reference spatiotemporal self-attention layer feature information;

[0102] The target video feature information set is determined based on the reference self-attention feature information and the reference spatiotemporal self-attention feature information.

[0103] In practical applications, reference self-attention feature information refers to feature values ​​that represent the relationship between pixels in the reference video, while reference spatiotemporal self-attention feature information refers to feature values ​​that represent the temporal and spatial relationships between entities in the reference video.

[0104] Specifically, reference self-attention feature information can be understood as the features of the relationships between pixels in the reference video itself. For example, in a reference video of a rabbit resting on the grass, where the rabbit's two eyes are pixels a and b, and pixel c is in the sky in the video, then pixels a and b are considered to have a strong relationship, while pixel c has a weak relationship with either pixel a or pixel b. This is reflected in the feature matrix as points...<a,b> point<b,a> Both are 0.9, points<c,a> ,point<c,b> All are 0.1, and the larger the number, the stronger the correlation.

[0105] The spatiotemporal self-attention feature information can be understood as the spatiotemporal relationships of various entities in the reference video. For example, in a reference video of a rabbit resting on grass, the rabbit itself changes over time, so its temporal relationship is strongly correlated. Since the rabbit is resting on the grass, the grass beneath it also moves as the rabbit moves, so the spatial relationship between the rabbit and the grass is strongly correlated. However, since the rabbit's movement does not affect the sky, and the sky is not related to time, the spatial and temporal relationships between the rabbit and the sky are weakly correlated. In other words, the spatiotemporal self-attention feature information can also be understood as the motion characteristics of entities in the reference video.

[0106] By acquiring the features that relate to each pixel in the video itself (which the user will not modify) and the motion features of entities in the video, and using these features to generate the target video, the generated target video can be made more similar to the unmodified parts of the aforementioned reference video.

[0107] Considering that the process of generating the reference video includes information on the initial prompt text and reference noise map features, the background of the target video can be kept consistent with that of the reference video in the subsequent process of generating the target video. Therefore, the reference self-attention feature information includes reference question feature information, reference key value feature information and reference value feature information, and the reference spatiotemporal self-attention feature information includes reference spatiotemporal feature information.

[0108] Furthermore, based on the reference self-attention feature information and the reference spatiotemporal self-attention feature information, a target video feature information set is determined, including:

[0109] Based on the reference self-attention feature information, determine the reference problem feature information and the reference key value feature information;

[0110] The reference spatiotemporal feature information is determined based on the aforementioned reference spatiotemporal self-attention feature information;

[0111] The reference problem feature information, the reference key value feature information, and the reference spatiotemporal feature information are determined as target feature information.

[0112] In practical applications, reference problem feature information and reference key value feature information are feature information used to calculate the relationship between each pixel in the reference video, while reference spatiotemporal feature information is feature information representing the temporal and spatial fusion features between entities in the reference video.

[0113] Specifically, the reference question feature information can be understood as using each pixel in the reference video as the feature information of the question, and the reference key value feature information can be understood as using each pixel in the reference video as the feature information of the key value. Multiplying the question feature information and the key value feature information by a dot product determines the relationship between the pixels in the reference video. Then, adding this to the answer value feature information of each pixel in the reference video strengthens the relationship between the pixels in the reference video. This allows us to obtain reference self-attention feature information representing the relationship between the pixels in the reference video.

[0114] Reference spatiotemporal feature information can be understood as the temporal and spatial relationships between various entities in the reference video, or as feature information representing the motion characteristics of the reference video.

[0115] By acquiring problem feature information and key value feature information of videos that users would not normally modify, it is possible to avoid the problem of the modification not being reflected when the user modifies the relationship between various pixels in the video. Furthermore, by acquiring the motion features of entities in the video and using these features to generate the target video, the generated target video can be made more similar to the unmodified parts of the aforementioned reference video.

[0116] Step 106: Receive video adjustment instructions and generate target prompt text based on the video adjustment instructions and the initial prompt text.

[0117] In practical applications, video adjustment commands are instructions to adjust the video. A video adjustment command can be understood as a user-sent instruction to adjust the video to be adjusted. For example, if a user wants to adjust the initial uploaded video, the video adjustment command is an instruction to adjust that initial video; if a user wants to adjust a reference video generated from text, the adjustment command is an instruction to adjust that reference video, and so on.

[0118] The target prompt text is the prompt text used to generate the target video. The target prompt text can be understood as text that is slightly modified from the initial text. For example, if the initial prompt text is "a video of a rabbit", the slightly modified target prompt text is "a video of a black rabbit".

[0119] In one embodiment provided in this specification, a video adjustment instruction carrying the user-edited "turn that rabbit black" is received, and the target prompt text "a video of a black rabbit" is generated by combining the previous initial prompt text "a rabbit photo video".

[0120] By combining the initial prompt text with the target prompt text generated, the initial prompt text can be slightly modified according to the user's requirements, so that the generated video is more similar to the unmodified parts of the original reference video.

[0121] Step 108: Input the reference noise map group, the target prompt text, and the target video feature information set into the second video generation model to obtain the target video generated by the second video generation model.

[0122] In practical applications, the second video generation model is the target for generating the target video, and the target video is generated based on the target prompt text.

[0123] Specifically, the second video generation model can be understood as a video generation model with the same structure and parameters as the first video generation model, used to generate the target video by combining a reference noise map group and target cue text. The target video can be understood as a video after modifying some entities or features in the reference video. For example, the original reference video is a white rabbit sleeping on the grass, and the modified target video is a black rabbit sleeping on the grass. The sleeping posture of the modified black rabbit and the original white rabbit, as well as the weather in the sky, are the same.

[0124] By using a second video generation model with the same parameters as the first video generation model to process the same set of reference noise maps, the modified target video can be made to have a certain degree of consistency with the original reference video. Then, by replacing some features of the reference video generated during the generation of the reference video with some features of the reference video, the generated target video can be made to be more similar to the unmodified parts of the original reference video.

[0125] Considering that two similar videos need to be generated, the second video generation model also includes a second feature extraction layer and a second video generation layer, and the parameters of the second video generation layer are the same as the parameters of the first video generation layer of the first video generation model.

[0126] Furthermore, the reference noise map group, the target prompt text, and the target video feature information set are input into the second video generation model to obtain the target video generated by the second video generation model, including:

[0127] The reference noise map group, the target prompt text, and the target video feature information set are input into the second feature extraction layer to obtain the target attention feature information set;

[0128] The target attention feature information set is input into the second video generation layer to obtain the target video output by the second video generation layer.

[0129] In practical applications, the second feature extraction layer is used to extract the features of the target video, the second video generation layer is used to generate a deep learning network layer for the target video based on the features extracted by the second feature extraction layer and some features extracted from the first feature extraction layer, and the target attention feature information set is used to input to the video generation layer to obtain the target video.

[0130] Specifically, the second feature extraction layer can be understood as a computational layer that generates the features of the target video. This layer can be a feature extraction layer with a neural network structure or a feature extraction layer without a neural network structure. This specification does not impose any restrictions on the results of this layer. If this layer is a feature extraction layer with a neural network structure, then the parameters of the neural network structure in this layer are the same as those of the first feature extraction layer mentioned above. For example, based on the input target prompt text and a set of noise maps, the self-attention features of the relationship between each pixel in the video to be generated, the cross-attention features of the relationship between the video to be generated and the target prompt text, etc., are extracted. This specification does not impose any restrictions on this.

[0131] The second video generation layer can be understood as a deep learning network layer that generates the corresponding video based on the extracted video features. The parameters of this neural network layer are the same as those of the first video generation layer mentioned above. For example, network models that can generate videos, such as DDIM and Static Diffusion, are used. This specification does not impose any restrictions on this.

[0132] The target attention feature information set can be understood as all the features of the target video to be generated. It includes the features extracted by the second feature extraction layer, and then the features in the target video feature set above are replaced with their corresponding features. For example, temporal attention features, spatial attention features, spatiotemporal attention features, self-attention features, cross-attention features, etc. This specification does not impose any restrictions on this.

[0133] The second feature extraction layer extracts some features of the target video. Then, the features that the user will not modify are replaced with the corresponding features obtained through the second feature extraction layer. The target video is generated by the replaced feature set, which makes the generated target video more similar to the unmodified parts of the reference video.

[0134] Furthermore, the reference noise map group, the target prompt text, and the target video feature information set are input into the second feature extraction layer to obtain the target attention feature information set, including:

[0135] The reference noise map group and the target prompt text are input into the second feature extraction layer to obtain the intermediate attention feature information set output by the second feature extraction layer, wherein the intermediate attention feature information set includes at least one attention feature information;

[0136] A target attention feature information set is generated based on the intermediate attention feature information set and the target video feature information set.

[0137] In practical applications, the intermediate attention feature information set is the set of all features output by the second feature extraction layer. Specifically, the intermediate attention feature information set can be understood as the features of the video generated solely based on the target prompt text, output by the second feature extraction layer, such as temporal attention features, spatial attention features, spatiotemporal attention features, self-attention features, cross-attention features, etc. This specification does not impose any limitations on this.

[0138] It should be noted that, considering the consistency between the generated target video and the original reference video in the unmodified parts, the problem feature information and key value problem feature information used to calculate the self-attention features of the target video can be replaced with the reference problem feature information and key value problem feature information of the reference video in the previous model.

[0139] By replacing the user-unmodifiable features (target video feature set) obtained above with the corresponding features obtained through the second feature extraction layer (features of the video generated solely based on the target prompt text), and then generating the target video based on the replaced feature set, compared to two videos generated by processing the same noisy graph group and prompt text with slight differences using only a model with the same parameters, the generated target video can be more similar to the unmodified parts of the reference video mentioned above.

[0140] Furthermore, based on the intermediate attention feature information set and the target video feature information set, a target attention feature information set is generated, including:

[0141] Determine target attention feature information, wherein the target attention feature information is any one of the attention feature information in the target video feature information set;

[0142] In the intermediate attention feature information set, attention feature information to be replaced is determined, wherein the attention feature information to be replaced is the attention feature information corresponding to the target attention feature information;

[0143] Replace each attention feature information to be replaced with the corresponding target attention feature information to generate a target attention feature information set.

[0144] In practical applications, the target attention feature information is the feature information in the target video feature information set, and the attention feature information to be replaced is the feature information corresponding to the target attention feature information.

[0145] Specifically, the attention feature information to be replaced can be understood as some features of the video generated solely based on the target prompt text. These features are usually not modified by the user. For example, self-attention features that represent the relationship between pixels in the video, spatiotemporal attention features that represent the motion relationship of entities in the video, etc. This specification does not impose any restrictions on them.

[0146] By finding the replacement feature information corresponding to each feature information in the target video feature information set and replacing the corresponding replacement feature information, and then generating the target video based on the replaced feature information set, compared to two videos generated by processing the same noisy image group and the prompt text with slight differences using the same model with the same parameters, the generated target video can be more similar to the unmodified parts of the above reference video.

[0147] Considering that the process of generating the reference video includes information on the initial prompt text and reference noise map features, the background of the target video can be kept consistent with that of the reference video in the subsequent process of generating the target video. Therefore, the intermediate attention feature information set includes intermediate question feature information, intermediate key value feature information and intermediate spatiotemporal feature information, and the target video feature information set includes reference question feature information, reference key value feature information and reference spatiotemporal feature information.

[0148] Furthermore, determining replacement attention feature information from the intermediate attention feature information set includes:

[0149] If the target attention feature information is the reference problem feature information, then the intermediate problem feature information corresponding to the reference problem feature information is determined to be the problem feature information to be replaced.

[0150] When the target attention feature information is the reference key value feature information, the intermediate key value feature information corresponding to the reference key value feature information is determined to be the key value feature information to be replaced.

[0151] When the target attention feature information is the reference spatiotemporal feature information, the intermediate spatiotemporal feature information corresponding to the reference spatiotemporal feature information is determined to be the spatiotemporal feature information to be replaced.

[0152] In practical applications, intermediate problem feature information and intermediate key value feature information are used to calculate the relationship between each pixel in the video generated based on the target prompt text. Problem feature information to be replaced and key value feature information to be replaced are used to replace the reference problem feature information and reference key value feature information. Intermediate spatiotemporal feature information is used to represent the temporal and spatial fusion features between entities in the video generated based on the target prompt text. Spatiotemporal feature information to be replaced is used to replace the reference spatiotemporal feature information.

[0153] Specifically, intermediate question feature information can be understood as using each pixel of the video generated based on the target prompt text as the feature information of the question, and intermediate key value feature information can be understood as using each pixel of the video generated based on the target prompt text as the feature information of the key value. Multiplying the feature information as the question and the feature information as the key value determines the relationship between each pixel in the video generated based on the target prompt text. Replacing the two intermediate question feature information and intermediate key value feature information above, the relationship between each pixel in the reference video can be obtained. Then, adding this to the value feature information of each pixel in the video generated based on the target prompt text as the answer, while ensuring that the relationship between each pixel in the generated target video is similar to that in the reference video, user modifications to the video (that is, the relationship between each pixel in the video generated based on the target prompt text) can be added. This achieves the goal of making the generated target video more similar to the unmodified parts of the reference video while adding some user modifications to the video, thereby improving the user experience.

[0154] Intermediate spatiotemporal feature information can be understood as the temporal and spatial relationships between various entities in a video generated based on target prompt text, or as feature information representing the motion features of a video generated based on target prompt text. By replacing the above intermediate spatiotemporal feature information with reference spatiotemporal feature information representing the motion features of a reference video, the motion information of various entities in the reference video can be effectively preserved in the newly generated target video, thereby making the generated target video more similar to the unmodified parts of the above reference video.

[0155] Furthermore, each attention feature to be replaced is replaced with the corresponding set of target attention features, including:

[0156] Replace the problem feature information to be replaced with the reference problem feature information;

[0157] Replace the key value feature information to be replaced with the reference key value feature information;

[0158] The spatiotemporal feature information to be replaced is replaced with the reference spatiotemporal feature information.

[0159] By replacing each feature in the target video feature information set with its corresponding feature information to be replaced, and then generating the target video based on the replaced feature information set, compared to generating two videos by processing the same noisy image group and the prompt text with slight differences using only a model with the same parameters, the generated target video can be more similar to the unmodified parts of the aforementioned reference video.

[0160] Considering that the accuracy of video modification can be further improved without taking resource consumption into account, the second feature extraction layer includes a second cross feature extraction unit;

[0161] Furthermore, after obtaining the target video generated by the second video generation model, the method further includes:

[0162] Based on the initial prompt text and the target prompt text, determine the prompt text distinguishing word information;

[0163] Based on the initial prompt text and the prompt text distinguishing word information, determine at least one related word information that is related to the prompt text distinguishing word information;

[0164] The second cross-feature extraction unit generates, based on the reference noise map group and the target prompt word text, the difference cross-attention feature information corresponding to the difference word information of the prompt word text and the association cross-attention feature information corresponding to each associated word information, which are generated by the second cross-feature extraction unit according to the reference noise map group and the target prompt word text;

[0165] Based on the distinguishing cross-attention feature information and each associated cross-attention feature information, as well as the reference video and the target video, a target edit video is generated.

[0166] In practical applications, the second cross-feature extraction unit is an attention unit that obtains the association between the target prompt text and the target video. The prompt text distinguishing word information is the word that distinguishes the target prompt text from the initial prompt text. The associated word information is the word in the initial prompt text that is related to the prompt text distinguishing word information. The distinguishing cross-attention feature information is the feature information that represents the association between the prompt text distinguishing word information and the target video. The association cross-attention feature information is the feature information that represents the association between each associated word and the target video.

[0167] Specifically, based on the initial prompt text and the target prompt text, the determined prompt text distinguishing word information can be understood as the difference between the initial prompt text and the target prompt text, or it can be further understood as the word that the user wants to adjust. For example, if the initial prompt text is "a rabbit photo video" and the target prompt text is "a black rabbit photo video", then the prompt text distinguishing word information corresponding to these two is "black".

[0168] The identified at least one related word information corresponding to the distinguishing word information of the prompt text can be understood as the number of words with the greatest correlation to the distinguishing word information of the prompt text in the initial prompt text, or more specifically, the number of words most relevant to the distinguishing word information of the prompt text. For example, if the initial prompt text is "a video of a rabbit photo shoot", and the identified distinguishing word information of the prompt text is "black", then if the number of associated words is 1, the associated word is "rabbit"; if the number of associated words is 2, the two associated words are "rabbit" and "photo shoot".

[0169] It should be noted that the method for obtaining the distinguishing words in the prompt text and the method for determining the related words corresponding to the distinguishing words in the prompt text can be by calculating the feature vectors of each word and calculating the distance, or by organizing the relationships between each word through a model, etc. This manual does not impose any restrictions on this method.

[0170] Distinguishing cross-attention feature information can be understood as the relationship between the obtained prompt text distinguishing words and the generated target video, or more specifically, the relationship between the prompt text distinguishing words and the various entities in the video. Similarly, associative cross-attention feature information can be understood as the relationship between the obtained associative words and the generated target video, or more specifically, the relationship between the associative words and the various entities in the video.

[0171] refer to Figure 4 , Figure 4 This specification illustrates a flowchart of a video editing process using an additional mask, according to an embodiment of this specification. The process involves receiving initial text information from a user and parsing it into standardized initial prompt text 404. A set of Gaussian noise reference noise maps 402 is then generated using preset rules. The initial prompt text 404 and the reference noise maps 402 are then input into the cross-feature extraction unit of the first feature extraction layer 4062 in the first video generation model 406 to obtain cross-feature information that integrates the noise map features and the initial prompt text features. This cross-feature information is then input into the self-attention extraction unit for self-attention extraction to obtain self-attention features that enhance key positions in the noise map. The cross-feature information is then input into the spatiotemporal self-attention extraction unit to obtain spatiotemporal feature information containing the spatiotemporal features of the reference noise maps. Finally, the cross-feature information, self-attention features, and spatiotemporal feature information are input into the first video generation layer 4064 to obtain a reference video 408.

[0172] After receiving the reference video 408, the user issues a modification command to the reference video 408, sending a video adjustment instruction. Then, based on the adjustment text in the video adjustment instruction, a target prompt text 416 with slight modifications to the initial prompt text 404 is generated. Next, the same reference noise map group 402 and target prompt text 416 are input into the second video generation model 418. Since the second video generation model 418 has the same structure and parameters as the first video generation model 406, the operations of the reference noise map group 402 and the target prompt text 416 in the second feature extraction layer 4184 and the first feature extraction layer 4062 are identical. Then, the problem feature information and key value feature information in the self-attention features generated based on the target prompt text 416 and the reference noise map group 402 are replaced with reference problem feature information 410 and reference key value feature information 414 generated based on the initial prompt text 404 and the reference noise map group 402. Then, the spatiotemporal feature information in the spatiotemporal feature information generated based on the target prompt text 416 and the reference noise map group 402 is replaced with reference spatiotemporal feature information 414 generated based on the initial prompt text 404 and the reference noise map group 402. The replaced feature information is input into the second video generation layer 4184 to obtain the target video 420 obtained by the user by modifying the reference video 408.

[0173] Subsequently, the distinct cross-attention feature information and the associated cross-attention feature information output by the cross-feature extraction unit in the second feature extraction layer 4182 are obtained. Then, the distinct cross-attention feature information and the associated cross-attention feature information are fused to obtain the target editing mask information 422. In addition, after noise processing of the reference video 408 and the target video 420, reference noise video 424 and target noise video 426 are obtained respectively. Then, the target editing mask information 422, reference noise video 424 and target noise video 426 are integrated to obtain the target editing video 428.

[0174] By extracting the cross-feature information corresponding to the distinguishing words between the target prompt text and the initial prompt text, as well as the cross-feature information corresponding to the related words of the distinguishing words, and then performing masking processing on the noisy video after noise processing of the reference video and the target video based on the extracted cross-feature information, and then generating the target edited video based on the result of the masking processing, the similarity between the unmodified parts of the target video and the reference video can be further improved.

[0175] Furthermore, based on the distinguishing cross-attention feature information and each associated cross-attention feature information, as well as the reference video and the target video, a target edited video is generated, including:

[0176] The distinguishing cross-attention feature information and each associated cross-attention feature information are processed according to the preset mask generation rules to generate target editing mask information;

[0177] According to preset noise processing rules, the reference video and the target video are processed to generate a reference noise video corresponding to the reference video and a target noise video corresponding to the target video;

[0178] A target editing video is generated based on the target editing mask information, the reference noise video, and the target noise video.

[0179] In practical applications, the mask generation rule is the data processing rule for generating target editing mask information, the reference noise video is a noise video carrying some features of the reference video, the target noise video is a noise video carrying some features of the target video, and the target editing video is the edited video generated according to the video adjustment instructions sent by the user.

[0180] Specifically, the mask generation rule can be understood as a rule that sets the regions related to the distinguishing words in the prompt text as masks. For example, the distinguishing cross-attention feature information and the various related cross-attention feature information are added together, and then regions whose sum is less than a preset threshold are set to 0. In other words, the smaller the value obtained by adding the distinguishing cross-attention feature information and the various related cross-attention feature information, the greater the correlation. Therefore, setting regions with values ​​less than the preset threshold to 0 is equivalent to setting the regions related to the distinguishing words in the prompt text as masks.

[0181] It should be noted that the noise processing rules here are the same as those used previously for converting the initial video into a reference noise map group, and will not be repeated here. Furthermore, the method for generating the target edit video can be as follows: first, combine the acquired mask information with the reference noise video and the target noise video to generate a masked noise video; then, restore this masked noise video using a mask restoration model to generate the target edit video. The aforementioned mask restoration model can be, for example, a VAE (Variational Autoencoder), etc., which restores a masked noise video to a video; this specification does not impose any restrictions on this.

[0182] In one embodiment provided in this specification, the specific method for combining the acquired mask information with the reference noise video and the target noise video to generate a masked noise video is shown in Formula 1:

[0183] x t =x src *(1-mask)+x dst *mask...Formula 1

[0184] Where, x t For masked noise video containing masking, x stc x is a reference noise video generated based on the reference video. dst This is a target noise video generated based on the target video.

[0185] By extracting the cross-feature information corresponding to the distinguishing words between the target prompt text and the initial prompt text, as well as the cross-feature information corresponding to the related words of the distinguishing words, and then performing masking processing on the noisy video after noise processing of the reference video and the target video based on the extracted cross-feature information, and then generating the target edited video based on the result of the masking processing, the similarity between the unmodified parts of the target video and the reference video can be further improved.

[0186] The scheme implemented in this specification generates a reference video from the initial prompt text, and uses the target video feature information set generated during the reference video generation process to generate a target video based on the target prompt text. Without needing to retrain the model, it ensures that the target video generated from the target prompt text, which is slightly modified from the initial prompt text, maintains a small difference from the reference video, thereby improving the accuracy of video editing.

[0187] Building upon this foundation, the basic video editing method only modifies self-attention features and spatiotemporal self-attention features, avoiding the mismatch issues caused by modifying cross-self-attention features. Further ensuring consistency between the generated target video and the reference video, the differences between the prompt text before and after modification are obtained, along with the feature mappings of the corresponding distinguishing words on the cross-self-attention layer. Based on this, the target video and reference video are further blended, ensuring that the target video only differs from the reference video in the modified areas, while remaining identical to the reference video in backgrounds unrelated to the modifications. This further improves the accuracy of video editing.

[0188] See Figure 5 , Figure 5 A flowchart of a video generation method applied to a cloud server according to an embodiment of this specification is shown, which specifically includes the following steps.

[0189] Step 502: Receive the video generation instruction sent by the terminal device, and generate initial prompt text according to the video generation instruction.

[0190] Step 504: Obtain a reference noise map group, input the reference noise map group and the initial prompt text into the first video generation model, and obtain the target video feature information set generated in the process of processing the reference noise map group and the initial prompt text.

[0191] Considering that when a user sends text, it is necessary to provide the user with feedback on the video generated based on the user's text, so that the user can decide on the parts to edit based on the previous video, the method further includes the following before obtaining the target video feature information set:

[0192] The reference noise map group and the initial prompt text are input into the first video generation model to obtain the reference video generated by the first video generation model;

[0193] If the video generation instruction carries initial text information, the reference video is returned to the end device.

[0194] When the user sends a video generation command that includes initial text, it can be understood that the user needs to generate a video based on the initial text information and then edit the video. Therefore, when the user sends a command that includes initial text, a reference video generated based on the initial prompt text needs to be returned to the user's device so that the user can send video adjustment commands to the server based on the reference video.

[0195] Step 506: Receive the video adjustment instruction sent by the terminal device, and generate the target prompt text according to the video adjustment instruction and the initial prompt text.

[0196] Step 508: Input the reference noise map group, the target prompt text, and the target video feature information set into the second video generation model, obtain the target video generated by the second video generation model, and return the target video to the end device.

[0197] Considering that users can further modify the target video if they are still not satisfied with the modified video, the method further includes the following after returning the target video to the terminal device:

[0198] The receiving end device receives a video adjustment instruction for the target video and generates a target adjusted video corresponding to the target video based on the video adjustment instruction.

[0199] The target adjustment video is sent to the end device.

[0200] In practical applications, a target adjustment video can be understood as a video generated based on a target video according to video adjustment instructions. By receiving video adjustment instructions generated by the user based on the target video and further adjusting the target video, users can make further modifications to the target video, thereby further improving the user experience.

[0201] The above is an illustrative scheme of the video generation method applied to a cloud server according to this embodiment. It should be noted that the technical solution of the video generation method applied to a cloud server belongs to the same concept as the technical solution of the video generation method described above. For details not described in detail in the technical solution of the video generation method applied to a cloud server, please refer to the description of the technical solution of the video generation method described above.

[0202] The solution described in this specification generates a reference video based on initial prompt text, and uses the target video feature information set generated during the reference video generation process to generate a target video based on the target prompt text. This ensures that the target video generated from the slightly modified target prompt text maintains minimal difference from the reference video, thereby improving the accuracy of video editing. Furthermore, the generated reference video is returned based on the user's text-based instructions, allowing the user on the client side to issue further editing instructions based on the generated reference video. Moreover, after completing one video edit, the user can edit the target video again, further enhancing the user experience.

[0203] See Figure 6 , Figure 6 This specification illustrates an architecture diagram of a video generation system according to one embodiment of the present specification. The video generation system may include a client 100 and a server 200.

[0204] Client 100 is used to send video generation and video editing instructions to server 200;

[0205] Server 200 is configured to receive video generation instructions sent by client 100 and generate initial prompt text according to the video generation instructions; acquire a reference noise map group, input the reference noise map group and the initial prompt text into a first video generation model to acquire a reference video generated by the first video generation model, and acquire target video feature values ​​generated by the first video generation model during the generation of the reference video; if the video generation instruction is an initial text instruction, send the reference video to client 100; then receive a video adjustment instruction sent by client 100 and generate target prompt text according to the video adjustment instruction and the initial prompt text; input the reference noise map group, the target prompt text, and the target video feature information set into a second video generation model to acquire the target video generated by the second video generation model; and send the target video to client 100.

[0206] Client 100 is also used to receive target videos and reference videos sent by server 200.

[0207] By applying the solution of the embodiments in this specification, a reference video is generated from the initial prompt text, and the set of target video feature information generated during the generation of the reference video is used to generate the target video based on the target prompt text. This ensures that the target video generated from the target prompt text, which is slightly modified from the initial prompt text, maintains a small difference from the reference video, thereby improving the accuracy of video editing.

[0208] The video generation system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the video generation scenario, server 200 is used to provide video generation services between multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0209] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In a video generation scenario, users can publish data streams to server 200 through client 100, and server 200 can generate target video and reference video based on the data stream, and push the target video and reference video to other clients that have established communication.

[0210] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0211] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices may have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0212] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0213] It is worth noting that the video generation method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the video generation method provided in the embodiments of this specification. In other embodiments, the video generation method provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0214] The following is in conjunction with the appendix Figure 7 Taking the video generation method provided in this specification as an example of its application in editing existing user videos, the video generation method will be further explained. Among other things, Figure 7This specification illustrates a flowchart of a process for editing an existing user video according to an embodiment, which includes the following steps.

[0215] Step 702: Receive video 1 sent by the user, and generate the initial prompt text "A rabbit photo video" based on video 1.

[0216] Step 704: Perform DDIM inversion on all frames in video 1 to obtain a set of potential noise images.

[0217] Step 706: Input the above-mentioned potential noise image group and the initial prompt text "a rabbit photo video" into the first video generation model, and obtain the reference problem feature value and reference key value feature value output by the self-attention layer in the first self-attention module of the video generation model, as well as the reference spatiotemporal key value feature value output by the spatiotemporal self-attention layer.

[0218] Step 708: Receive the adjustment command sent by the user: "Change the rabbit in video 1 to black".

[0219] Step 710: Based on the above adjustment instructions, generate the adjustment prompt text "A black rabbit photo video".

[0220] Step 712: Input the above-mentioned potential noise image group and the adjustment prompt text "a black rabbit photo video" into the second video generation model. Replace the above-mentioned reference problem feature value, reference key value feature value and reference spatiotemporal key value feature value with the intermediate problem feature value and intermediate key value feature value output by the self-attention layer of the second self-attention module and the intermediate spatiotemporal key value feature value output by the spatiotemporal self-attention layer in the second video generation model to obtain the target self-attention feature value after replacement.

[0221] Step 714: Input the target self-attention feature value into the diffusion module of the second video generation model to obtain the target video.

[0222] The scheme implemented in this specification generates a reference video from the initial prompt text, and uses the target video feature information set generated during the reference video generation process to generate a target video based on the target prompt text. Without needing to retrain the model, it ensures that the target video generated from the target prompt text, which is slightly modified from the initial prompt text, maintains a small difference from the reference video, thereby improving the accuracy of video editing.

[0223] Based on this, the basic video modification method only modifies the self-attention features and spatiotemporal self-attention features, avoiding the mismatch problem caused by modifying the cross-self-attention features. Further ensuring the consistency between the generated target video and the reference video, by obtaining the differences between the prompt text before and after modification and obtaining the feature mapping of the corresponding distinguishing words on the cross-self-attention layer, the target video and the reference video are further blended. This ensures that the target video only shows differences from the reference video in the modified areas, while remaining identical to the reference video in backgrounds unrelated to the modifications, thereby further improving the accuracy of video editing. Corresponding to the above method embodiments, this specification also provides embodiments of a video generation device. Figure 8 A schematic diagram of a video generation apparatus provided in one embodiment of this specification is shown.

[0224] like Figure 8 As shown, the device includes:

[0225] The generation instruction receiving module 802 is configured to receive video generation instructions and generate initial prompt text according to the video generation instructions;

[0226] The video feature value acquisition module 804 is configured to acquire a reference noise map group, input the reference noise map group and the initial prompt text into the first video generation model, and acquire the target video feature value generated during the processing of the reference noise map group and the initial prompt text;

[0227] The adjustment instruction receiving module 806 is configured to receive video adjustment instructions and generate target prompt text based on the video adjustment instructions and the initial prompt text;

[0228] The target video generation module 808 is configured to input the reference noise map group, the target prompt text, and the target video feature information set into the second video generation model to obtain the target video generated by the second video generation model.

[0229] Optionally, the video generation instruction carries initial text information;

[0230] The generation instruction receiving module 802 is further configured to:

[0231] Parse the video generation instructions to obtain the initial text information;

[0232] The initial prompt text is generated based on the initial text information;

[0233] Accordingly, a reference noise map set is obtained, including:

[0234] The reference noise map group is generated by using preset noise map group generation rules;

[0235] Accordingly, receive video adjustment instructions, including:

[0236] Receive the video adjustment instruction corresponding to the reference video.

[0237] Optionally, the video generation instruction carries initial video information;

[0238] The generation instruction receiving module 802 is further configured to:

[0239] Parse the video generation instruction to obtain the initial video information;

[0240] The initial prompt text is generated based on the initial video information, wherein the initial prompt text is used to describe the initial video information;

[0241] Accordingly, a reference noise map set is obtained, including:

[0242] Based on the initial video information, the reference noise map group corresponding to the initial video information is generated;

[0243] Accordingly, receive video adjustment instructions, including:

[0244] Receive the video adjustment instruction corresponding to the initial video information.

[0245] Optionally, the video feature value acquisition module 804 is further configured to:

[0246] The reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video;

[0247] Obtain the set of target video feature information generated by the first video generation model during the process of generating the reference video.

[0248] Optionally, the first video generation model includes a first feature extraction layer and a first video generation layer;

[0249] The video feature value acquisition module 804 is further configured to:

[0250] The reference noise map group and the initial prompt text are input into the first feature extraction layer to obtain the reference attention feature information set output by the first feature extraction layer;

[0251] The reference attention feature information set is input into the first video generation layer to obtain the reference video output by the first video generation layer.

[0252] Optionally, the video feature value acquisition module 804 is further configured to:

[0253] Obtain a set of reference attention feature information output by the first feature extraction layer, wherein the set of reference attention feature information includes at least one set of attention feature information;

[0254] A target video feature information set is determined based on the reference attention feature information set, wherein the target video feature information set includes at least one attention feature information.

[0255] Optionally, the reference attention feature information set includes reference self-attention feature information and reference spatiotemporal self-attention feature information;

[0256] The video feature value acquisition module 804 is further configured to:

[0257] Based on the aforementioned set of reference attention feature information, determine the reference self-attention layer feature information and the reference spatiotemporal self-attention layer feature information;

[0258] The target video feature information set is determined based on the reference self-attention feature information and the reference spatiotemporal self-attention feature information.

[0259] Optionally, the reference self-attention feature information includes reference problem feature information, reference key value feature information, and reference value feature information, and the reference spatiotemporal self-attention feature information includes reference spatiotemporal feature information;

[0260] The video feature value acquisition module 804 is further configured to:

[0261] Based on the reference self-attention feature information, determine the reference problem feature information and the reference key value feature information;

[0262] The reference spatiotemporal feature information is determined based on the aforementioned reference spatiotemporal self-attention feature information;

[0263] The reference problem feature information, the reference key value feature information, and the reference spatiotemporal feature information are determined as target feature information.

[0264] Optionally, the second video generation model includes a second feature extraction layer and a second video generation layer, wherein the parameters of the second video generation layer are the same as the parameters of the first video generation layer of the first video generation model.

[0265] The target video generation module 808 is further configured as follows:

[0266] The reference noise map group, the target prompt text, and the target video feature information set are input into the second feature extraction layer to obtain the target attention feature information set;

[0267] The target attention feature information set is input into the second video generation layer to obtain the target video output by the second video generation layer.

[0268] Optionally, the target video generation module 808 is further configured to:

[0269] The reference noise map group and the target prompt text are input into the second feature extraction layer to obtain the intermediate attention feature information set output by the second feature extraction layer, wherein the intermediate attention feature information set includes at least one attention feature information;

[0270] A target attention feature information set is generated based on the intermediate attention feature information set and the target video feature information set.

[0271] Optionally, the target video generation module 808 is further configured to:

[0272] Determine target attention feature information, wherein the target attention feature information is any one of the attention feature information in the target video feature information set;

[0273] In the intermediate attention feature information set, attention feature information to be replaced is determined, wherein the attention feature information to be replaced is the attention feature information corresponding to the target attention feature information;

[0274] Replace each attention feature information to be replaced with the corresponding target attention feature information to generate a target attention feature information set.

[0275] Optionally, the intermediate attention feature information set includes intermediate problem feature information, intermediate key value feature information, and intermediate spatiotemporal feature information, and the target video feature information set includes reference problem feature information, reference key value feature information, and reference spatiotemporal feature information;

[0276] The target video generation module 808 is further configured as follows:

[0277] If the target attention feature information is the reference problem feature information, then the intermediate problem feature information corresponding to the reference problem feature information is determined to be the problem feature information to be replaced.

[0278] When the target attention feature information is the reference key value feature information, the intermediate key value feature information corresponding to the reference key value feature information is determined to be the key value feature information to be replaced.

[0279] When the target attention feature information is the reference spatiotemporal feature information, the intermediate spatiotemporal feature information corresponding to the reference spatiotemporal feature information is determined to be the spatiotemporal feature information to be replaced.

[0280] Optionally, the target video generation module 808 is further configured to:

[0281] Replace the problem feature information to be replaced with the reference problem feature information;

[0282] Replace the key value feature information to be replaced with the reference key value feature information;

[0283] Replace the spatiotemporal feature information to be replaced with the reference spatiotemporal feature information.

[0284] Optionally, the second feature extraction layer includes a second cross-feature extraction unit;

[0285] Optionally, the video generation apparatus further includes a mask module, configured as follows:

[0286] Based on the initial prompt text and the target prompt text, determine the prompt text distinguishing word information;

[0287] Based on the initial prompt text and the prompt text distinguishing word information, determine at least one related word information that is related to the prompt text distinguishing word information;

[0288] The second cross-feature extraction unit generates, based on the reference noise map group and the target prompt word text, the difference cross-attention feature information corresponding to the difference word information of the prompt word text and the association cross-attention feature information corresponding to each associated word information, which are generated by the second cross-feature extraction unit according to the reference noise map group and the target prompt word text;

[0289] Based on the distinguishing cross-attention feature information and each associated cross-attention feature information, as well as the reference video and the target video, a target edit video is generated.

[0290] The optional mask module is further configured as follows:

[0291] The distinguishing cross-attention feature information and each associated cross-attention feature information are processed according to the preset mask generation rules to generate target editing mask information;

[0292] According to preset noise processing rules, the reference video and the target video are processed to generate a reference noise video corresponding to the reference video and a target noise video corresponding to the target video;

[0293] A target editing video is generated based on the target editing mask information, the reference noise video, and the target noise video.

[0294] The above is an illustrative scheme of a video generation apparatus according to this embodiment. It should be noted that the technical solution of this video generation apparatus and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the video generation apparatus, please refer to the description of the technical solution of the video generation method described above.

[0295] The scheme implemented in this specification generates a reference video from the initial prompt text, and uses the target video feature information set generated during the reference video generation process to generate a target video based on the target prompt text. Without needing to retrain the model, it ensures that the target video generated from the target prompt text, which is slightly modified from the initial prompt text, maintains a small difference from the reference video, thereby improving the accuracy of video editing.

[0296] Building upon this foundation, the basic video editing method only modifies self-attention features and spatiotemporal self-attention features, avoiding the mismatch issues caused by modifying cross-self-attention features. Further ensuring consistency between the generated target video and the reference video, the differences between the prompt text before and after modification are obtained, along with the feature mappings of the corresponding distinguishing words on the cross-self-attention layer. Based on this, the target video and reference video are further blended, ensuring that the target video only differs from the reference video in the modified areas, while remaining identical to the reference video in backgrounds unrelated to the modifications. This further improves the accuracy of video editing.

[0297] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0298] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0299] In one embodiment of this specification, the aforementioned components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0300] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0301] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the video generation method described above or the video generation method applied to a cloud server.

[0302] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the video generation method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the video generation method or the video generation method applied to cloud servers.

[0303] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the video generation method described above or the video generation method applied to a cloud server.

[0304] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the video generation method or the video generation method applied to a cloud server described above. Details not described in detail in the technical solution of the storage medium can be found in the description of the video generation method or the video generation method applied to a cloud server described above.

[0305] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the video generation method described above or the video generation method applied to a cloud server.

[0306] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the aforementioned video generation method or the video generation method applied to a cloud server. Details not described in detail in the computer program's technical solution can be found in the descriptions of the aforementioned video generation method or the video generation method applied to a cloud server.

[0307] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0308] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0309] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0310] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0311] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A video generation method, comprising: Receive video generation instructions and generate initial prompt text based on the video generation instructions; A reference noise map group is obtained, and the reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video. The target video feature information set generated by the first video generation model in the process of generating the reference video is obtained, wherein the target video feature information set includes reference question feature information, reference key value feature information and reference spatiotemporal feature information. Receive video adjustment instructions, and generate target prompt text based on the video adjustment instructions and the initial prompt text; The reference noise map group, the target prompt text, and the target video feature information set are input into the second video generation model to obtain the target video generated by the second video generation model.

2. The method as described in claim 1, wherein the first video generation model comprises a first feature extraction layer and a first video generation layer; The reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video, including: The reference noise map group and the initial prompt text are input into the first feature extraction layer to obtain the reference attention feature information set output by the first feature extraction layer; The reference attention feature information set is input into the first video generation layer to obtain the reference video output by the first video generation layer.

3. The method as described in claim 2, wherein obtaining the target video feature information set generated by the first video generation model during the generation of the reference video includes: Obtain a set of reference attention feature information output by the first feature extraction layer, wherein the set of reference attention feature information includes at least one set of attention feature information; A target video feature information set is determined based on the reference attention feature information set, wherein the target video feature information set includes at least one attention feature information.

4. The method as described in claim 3, wherein the set of reference attention feature information includes reference self-attention feature information and reference spatiotemporal self-attention feature information; Determining the target video feature information set based on the reference attention feature information set includes: Based on the aforementioned set of reference attention feature information, determine the reference self-attention layer feature information and the reference spatiotemporal self-attention layer feature information; The target video feature information set is determined based on the reference self-attention feature information and the reference spatiotemporal self-attention feature information.

5. The method as described in claim 4, wherein the reference self-attention feature information includes reference problem feature information, reference key value feature information, and reference value feature information, and the reference spatiotemporal self-attention feature information includes reference spatiotemporal feature information; Based on the reference self-attention feature information and the reference spatiotemporal self-attention feature information, a target video feature information set is determined, including: Based on the reference self-attention feature information, determine the reference problem feature information and the reference key value feature information; The reference spatiotemporal feature information is determined based on the aforementioned reference spatiotemporal self-attention feature information; The reference problem feature information, the reference key value feature information, and the reference spatiotemporal feature information are determined as target feature information.

6. The method as described in claim 1, wherein the second video generation model includes a second feature extraction layer and a second video generation layer, and the parameters of the second video generation layer are the same as the parameters of the first video generation layer of the first video generation model; The reference noise map group, the target prompt text, and the target video feature information set are input into the second video generation model to obtain the target video generated by the second video generation model, including: The reference noise map group, the target prompt text, and the target video feature information set are input into the second feature extraction layer to obtain the target attention feature information set; The target attention feature information set is input into the second video generation layer to obtain the target video output by the second video generation layer.

7. The method of claim 6, wherein the reference noise map group, the target cue text, and the target video feature information set are input into the second feature extraction layer to obtain the target attention feature information set, comprising: The reference noise map group and the target prompt text are input into the second feature extraction layer to obtain the intermediate attention feature information set output by the second feature extraction layer, wherein the intermediate attention feature information set includes at least one attention feature information; A target attention feature information set is generated based on the intermediate attention feature information set and the target video feature information set.

8. The method of claim 7, wherein generating a target attention feature information set based on the intermediate attention feature information set and the target video feature information set comprises: Determine target attention feature information, wherein the target attention feature information is any one of the attention feature information in the target video feature information set; In the intermediate attention feature information set, attention feature information to be replaced is determined, wherein the attention feature information to be replaced is the attention feature information corresponding to the target attention feature information; Replace each attention feature information to be replaced with the corresponding target attention feature information to generate a target attention feature information set.

9. The method as described in claim 8, wherein the intermediate attention feature information set includes intermediate problem feature information, intermediate key value feature information, and intermediate spatiotemporal feature information; Determining the attention feature information to be replaced from the intermediate attention feature information set includes: If the target attention feature information is the reference problem feature information, then the intermediate problem feature information corresponding to the reference problem feature information is determined to be the problem feature information to be replaced. When the target attention feature information is the reference key value feature information, the intermediate key value feature information corresponding to the reference key value feature information is determined to be the key value feature information to be replaced. When the target attention feature information is the reference spatiotemporal feature information, the intermediate spatiotemporal feature information corresponding to the reference spatiotemporal feature information is determined to be the spatiotemporal feature information to be replaced.

10. The method of claim 9, wherein each attention feature information to be replaced is replaced with the corresponding target attention feature information, comprising: Replace the problem feature information to be replaced with the reference problem feature information; Replace the key value feature information to be replaced with the reference key value feature information; Replace the spatiotemporal feature information to be replaced with the reference spatiotemporal feature information.

11. The method of claim 6, wherein the second feature extraction layer includes a second cross feature extraction unit; After obtaining the target video generated by the second video generation model, the method further includes: Based on the initial prompt text and the target prompt text, determine the prompt text distinguishing word information; Based on the initial prompt text and the prompt text distinguishing word information, determine at least one related word information corresponding to the prompt text distinguishing word information; The second cross-feature extraction unit obtains the difference cross-attention feature information corresponding to the difference word information of the prompt text and the association cross-attention feature information corresponding to each associated word information, generated by the second cross-feature extraction unit based on the reference noise map group and the target prompt text; Based on the distinguishing cross-attention feature information and each associated cross-attention feature information, as well as the reference video and the target video, a target edit video is generated.

12. The method of claim 11, wherein generating a target edited video based on the distinguishing cross-attention feature information and each associated cross-attention feature information, as well as the reference video and the target video, comprises: The distinguishing cross-attention feature information and each associated cross-attention feature information are processed according to the preset mask generation rules to generate target editing mask information; According to preset noise processing rules, the reference video and the target video are processed to generate a reference noise video corresponding to the reference video and a target noise video corresponding to the target video; A target editing video is generated based on the target editing mask information, the reference noise video, and the target noise video.

13. The method as described in claim 1, wherein the video generation instruction carries initial text information; Receive a video generation instruction and generate initial prompt text based on the video generation instruction, including: Parse the video generation instructions to obtain the initial text information; The initial prompt text is generated based on the initial text information; Accordingly, a reference noise map set is obtained, including: The reference noise map group is generated by using preset noise map group generation rules; Accordingly, receive video adjustment instructions, including: Receive the video adjustment instruction corresponding to the reference video.

14. The method as described in claim 1, wherein the video generation instruction carries initial video information; Receive a video generation instruction and generate initial prompt text based on the video generation instruction, including: Parse the video generation instruction to obtain the initial video information; The initial prompt text is generated based on the initial video information, wherein the initial prompt text is used to describe the initial video information; Accordingly, a reference noise map set is obtained, including: Based on the initial video information, the reference noise map group corresponding to the initial video information is generated; Accordingly, receive video adjustment instructions, including: Receive the video adjustment instruction corresponding to the initial video information.

15. A video generation method, applied to a cloud server, comprising: The receiving end device sends a video generation instruction, and generates an initial prompt text based on the video generation instruction; A reference noise map group is obtained, and the reference noise map group and the initial prompt text are input into the first video generation model to generate a reference video. The target video feature information set generated by the first video generation model in the process of generating the reference video is obtained, wherein the target video feature information set includes reference question feature information, reference key value feature information and reference spatiotemporal feature information. The receiver sends a video adjustment command from the receiving end device, and generates a target prompt text based on the video adjustment command and the initial prompt text; The reference noise map set, the target prompt text, and the target video feature information set are input into the second video generation model to obtain the target video generated by the second video generation model, and the target video is returned to the end device.

16. The method of claim 15, further comprising, before acquiring the target video feature information set: If the video generation instruction carries initial text information, the reference video is returned to the end device.

17. The method of claim 15, further comprising, after returning the target video to the end-side device: The receiving end device receives a video adjustment instruction for the target video and generates a target adjusted video corresponding to the target video based on the video adjustment instruction. The target adjustment video is sent to the end device.

18. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-14 or 15-17.

19. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-14 or 15-17.

20. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-14 or 15-17.

Images