Multi-scene video generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a multi-scene video generation method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: encoding text information when prompt information inputted by a user is text information containing a first number of scenes, and obtaining a first number group of text features, cascading with the first number group of original noise information according to a time sequence to obtain cascading features; inputting the cascade features into an attention layer, and performing attention mask processing on the cascade features to obtain weighted features; taking features corresponding to the original noise information in the weighted features as predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain denoised features; and decoding the de-noising feature to obtain video information containing the first number of scenes. The video information containing the preset number of scenes is generated based on the attention mask, the video clips of different scenes are generated at the same time, and the natural degree of transition between the scenes is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of video generation technology, and in particular to a multi-scene video generation method, device, electronic device, storage medium and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Multi-scene video refers to a video that contains multiple different scenes, and these scenes may be switched in time or space.

[0004] Multi-scene video generation refers to generating videos from multi-scene descriptions while maintaining semantic consistency and temporal order among scenes.

[0005] When generating a multi-scene video, the related technology usually generates multiple single-scene videos first, and then splices the multiple single-scene videos together to obtain the multi-scene video.

[0006] However, this approach often leaves noticeable seams between scenes, resulting in unnatural transitions between scenes in the video. Summary of the invention

[0007] In view of this, the purpose of the present disclosure is to propose a multi-scene video generation method, device, electronic device, storage medium and program product, which at least to a certain extent solve one of the technical problems in the related art.

[0008] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a multi-scene video generation method, including:

[0009] Determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0010] In response to the analysis result that the prompt information is text information containing a first number of scenes, the text information is encoded to obtain a first number of groups of text features, and the first number of groups of text features are cascaded with the first number of groups of original noise information in chronological order to obtain cascade features;

[0011] Inputting the cascade features into an attention layer, and performing attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0012] Taking the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0013] The denoising feature is decoded to obtain video information including a first number of scenes.

[0014] Based on the same inventive concept, a second aspect of the exemplary embodiment of the present disclosure provides a multi-scene video generation device, including:

[0015] A prompt information analysis module is configured to determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0016] The encoding and noise adding module is configured to, in response to the analysis result that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and cascade the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features;

[0017] An attention mask module is configured to input the cascade features into an attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0018] a denoising and restoration module, configured to use the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0019] The decoding module is configured to decode the denoising feature to obtain video information containing a first number of scenes.

[0020] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0021] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0022] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0023] From the above, it can be seen that the multi-scene video generation method, device, electronic device, storage medium and program product provided by the embodiment of the present disclosure include: determining the prompt information input by the user, analyzing the prompt information, and obtaining the analysis result; in response to the analysis result that the prompt information is text information containing a first number of scenes, encoding the text information to obtain a first number of groups of text features, cascading the first number of groups of text features with the first number of groups of original noise information in time order to obtain cascade features; inputting the cascade features into the attention layer, performing attention mask processing on the cascade features based on the attention layer to obtain weighted features; using the features corresponding to the original noise information in the weighted features as predicted noise information, denoising the original noise information based on the predicted noise information to obtain denoised features; decoding the denoised features to obtain video information containing the first number of scenes. The present disclosure generates video information containing a preset number of scenes based on the attention mask. Since video clips of different scenes are generated at the same time, no obvious seams are left between the scenes, which improves the naturalness of the transition between scenes in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of an application scenario of the multi-scenario video generation method provided by an exemplary embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of a flow chart of a multi-scene video generation method provided by an exemplary embodiment of the present disclosure;

[0027] Figure 3 Another flowchart of a multi-scene video generation method provided by an exemplary embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram of a training method for an attention layer provided for an exemplary embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram of a group attention mechanism provided for an exemplary embodiment of the present disclosure;

[0030] Figure 6 Another schematic diagram of a group attention mechanism provided for an exemplary embodiment of the present disclosure;

[0031] Figure 7 A schematic diagram of a structure of a multi-scene video generation device provided by an exemplary embodiment of the present disclosure;

[0032] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0037] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0038] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0041] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0042] As described in the background art, multi-scene video refers to a video content that includes multiple different scenes, and these scenes may be switched in time or space.

[0043] Multi-scene video generation refers to generating videos from multi-scene descriptions while maintaining semantic consistency and temporal order among scenes.

[0044] However, the inventors of the present disclosure have discovered that connecting these key frames often results in unnatural motion transitions because the key frame sampling process fails to fully consider the time positioning and related motion dynamics in the multi-scene video timeline.

[0045] In order to solve the above problems, the present disclosure provides a multi-scene video generation solution, which specifically includes: when the prompt information input by the user is text information containing a first number of scenes, encoding the text information to obtain a first number of groups of text features, and cascading them with the first number of groups of original noise information in chronological order to obtain cascade features; inputting the cascade features into an attention layer, performing attention masking on the cascade features to obtain weighted features; using the features in the weighted features corresponding to the original noise information as predicted noise information, and denoising the original noise information based on the predicted noise information to obtain denoised features; decoding the denoised features to obtain video information containing the first number of scenes.

[0046] The present invention generates video information containing a preset number of scenes based on an attention mask, and simultaneously generates video clips of different scenes, thereby improving the naturalness of transitions between scenes.

[0047] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0048] refer to Figure 1 , which is a schematic diagram of an application scenario of the multi-scenario video generation method provided by an exemplary embodiment of the present disclosure.

[0049] This application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, the server 102, and the data storage system 103 may be connected via a wired or wireless communication network to achieve data interaction.

[0050] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0051] Server 102 and data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0052] In some exemplary embodiments, the multi-scene video generation method may be run on the terminal device 101 or the server 102 .

[0053] When the multi-scene video generation method runs on the server 102, the server 102 is used to provide a multi-scene video generation service to a user of the terminal device 101. The terminal device 101 is installed with a client that communicates with the server 102. The user can input prompt information through the client, and the client sends the prompt information to the server 102. The server 102 determines the prompt information input by the user, analyzes the prompt information, and obtains the analysis result.

[0054] When the analysis result is a prompt message including text information of the first number of scenes:

[0055] The server 102 encodes the text information to obtain a first number of groups of text features, and cascades the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features; inputs the cascade features into an attention layer, and performs attention masking on the cascade features based on the attention layer to obtain weighted features; uses the features in the weighted features corresponding to the original noise information as predicted noise information, and denoises the original noise information based on the predicted noise information to obtain denoised features; and decodes the denoised features to obtain video information containing a first number of scenes.

[0056] When the analysis result is that the prompt information is text information containing the second number of scenes and video information containing the third number of scenes:

[0057] The server 102 encodes the text information to obtain a second number group of text features, encodes the video information to obtain a third number group of video features, and cascades the second number group of text features, the third number group of video features, and a group of original noise information in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one; inputs the cascade features into the attention layer, performs attention masking on the cascade features based on the attention layer, and obtains weighted features; uses the features in the weighted features corresponding to the original noise information as predicted noise information, and denoises the original noise information based on the predicted noise information to obtain denoised features; decodes the denoised features to obtain video information containing a scene.

[0058] After generating video information containing one scene, the server 102 may send the video information containing one scene to the client, and the client displays the video information containing one scene to the user; the server 102 may also splice the video information containing one scene after the video information containing a third number of scenes input by the user and send it to the client, and the client displays the spliced ​​video information to the user.

[0059] A large amount of training data is stored in the data storage system 103. The server 102 can train the attention layer based on the large amount of training data so that the attention layer can predict the input features based on the attention mask processing. The source of the training data includes but is not limited to an existing database, data crawled from the Internet, or data uploaded by the user when using the client. When the accuracy of the output of the attention layer reaches a certain requirement, the server 102 can provide the user with a multi-scene video generation service based on the attention layer. At the same time, the server 102 can also continuously optimize the attention layer based on the newly added training data.

[0060] Combine the following Figure 1The training method of the attention layer and the multi-scene video generation method according to the exemplary embodiment of the present disclosure are described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0061] The present disclosure has at least two application scenarios, one is to generate a multi-scene video based on a text containing multiple scenes input by a user, and the other is to expand the input video based on a video containing multiple scenes input by a user and a text for expansion to obtain a video containing more scenes. Below, the application scenario of generating a multi-scene video based on a text containing multiple scenes input by a user is first introduced:

[0062] refer to Figure 2 , which is a schematic diagram of an application scenario of the multi-scenario video generation method provided by an exemplary embodiment of the present disclosure.

[0063] The multi-scene video generation method comprises the following steps:

[0064] Step S210: determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result.

[0065] In this disclosure, the analysis results include at least the following two situations:

[0066] (1) The analysis result is that the prompt information is text information containing a first number of scenes, and optionally, the first number is greater than or equal to three.

[0067] (2) The analysis result is that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0068] The present disclosure provides Figure 2 The embodiment shown (i.e., this embodiment) is directed to situation (1), where Figure 3 The embodiment shown (ie, the next embodiment) is directed to situation (2).

[0069] The present disclosure provides a graphical user interface for users to input text information and video information.

[0070] During specific implementation, the user may also input text information through voice input.

[0071] Step S220: In response to the analysis result that the prompt information is text information containing a first number of scenes, the text information is encoded to obtain a first number of groups of text features, and the first number of groups of text features are cascaded with the first number of groups of original noise information in chronological order to obtain cascade features.

[0072] In this exemplary embodiment, the first number is greater than or equal to three, that is, the text information describes contents about at least three scenes.

[0073] In this exemplary embodiment, in order to improve the richness of the generated multi-scene video, before encoding the text information, the method further includes:

[0074] Expand the text information based on a large language model to obtain expanded text information;

[0075] Then, encoding the text information includes:

[0076] The expanded text information is encoded.

[0077] In a specific implementation, each scene in the text information containing the first number of scenes is expanded respectively.

[0078] It should be noted that the expanded text information still only includes the first number of scenes, and the number of scenes is not expanded.

[0079] In this exemplary embodiment, the text information is encoded to obtain a first number of groups of text features, wherein each group of text features corresponds to a scene.

[0080] In this exemplary embodiment, the original noise information is random noise.

[0081] It should be noted that the number of groups of original noise information is the same as the number of scenes, and is used to finally generate video clips having the same number of scenes.

[0082] In this exemplary embodiment, the first number group of text features and the first number group of original noise information are cascaded in time order to obtain cascade features, where:

[0083] The first number group of text features is located in the front, and the first number group of original noise information is located in the back.

[0084] Step S230: input the cascade features into the attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features.

[0085] Among them, considering that directly predicting based on cascade features will destroy the fine-grained alignment relationship between the original scene and its corresponding text annotation, the present disclosure introduces an attention mask to establish such a fine-grained one-to-one alignment, so that each scene fragment can focus on its corresponding text annotation more effectively.

[0086] In this exemplary embodiment, attention mask processing is performed on the cascade features based on the attention layer.

[0087] In specific implementation, the attention layer includes a 3D full attention layer, the core of which is the Self-Attention mechanism, which allows the 3D full attention layer to dynamically focus on different parts of the input features when processing data.

[0088] The attention mechanism is controlled by creating a sparse mask so that the 3D full attention layer only focuses on specific parts of the input features.

[0089] By defining a mask function, the function returns a Boolean mask according to the input query and the index position of the key, indicating which positions of the keys the query should focus on. For example, when the query is a text feature, each query is set to focus only on the text features and video features belonging to the same scene (when there is no video feature, it is replaced by the original noise information); when the query is a video feature (when there is no video feature, it is replaced by the original noise information), each query is set to focus only on the text features and all video features belonging to the same scene (when there is no video feature, it is replaced by the original noise information).

[0090] As an example, see Figure 4 , where “0, 1”, “2, 3”, and “4, 5” are three groups of text features, and “6, 7, 8”, “9, A, B”, and “C, D, E” are three groups of video features (when there is no video feature, it is replaced by the original noise information). Among them, “0, 1” is the text description of “6, 7, 8”, “2, 3” is the text description of “9, A, B”, and “4, 5” is the text description of “C, D, E”.

[0091] When the mask is not used, the prediction of “6, 7, 8” will not only focus on its text description “0, 1”, but also on the text descriptions “2, 3” and “4, 5”, which will obviously affect the accuracy of the generated video clip.

[0092] Through Figure 4 The symmetrical binary attention mask shown in the figure only pays attention to the text descriptions “0, 1” when predicting “6, 7, 8”, and does not pay attention to the text descriptions “2, 3” and “4, 5”, which improves the accuracy of the generated video clips.

[0093] Through the above exemplary embodiments, each scene segment can be more effectively focused on its corresponding text annotation, thereby improving the accuracy of the generated video segment.

[0094] However, the inventors of the present disclosure have found that in the above exemplary embodiments, due to the long total length of the sequence of cascade features, the memory usage of the attention mask is significantly increased. In order to solve this problem, the present disclosure also provides a group attention mechanism, specifically:

[0095] In this exemplary embodiment, performing attention mask processing on the cascade features based on the attention layer includes:

[0096] Dividing the cascade features into the first number group of text features and the first number group of original noise information;

[0097] The text feature and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0098] When implementing it, combine Figure 4 The symmetrical binary attention mask shown in the figure is used to illustrate that the entire one-dimensional cascade feature sequence "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E" is divided into 2n (n=3) groups, including 3 text feature groups "0, 1", "2, 3", "4, 5" and 3 video feature groups (when there is no video feature, it is replaced by the original noise information) "6, 7, 8", "9, A, B", "C, D, E".

[0099] Each set of features is considered as a query parameter, and the corresponding key parameter and value parameter are found in the attention mask, followed by a cross-attention calculation.

[0100] As an example, see Figure 5 , take a set of text features “0, 1” as an example, and use the text feature set “0, 1” as the query parameter W Q , determine the corresponding key parameter W in the attention mask K Sum parameter W V , perform cross attention calculation.

[0101] Among them, when the text feature group "0, 1" is used as the query parameter W Q When the key parameter W K Sum parameter W V It is composed of the text feature group “0, 1” and the video feature group “6, 7, 8” of the scene corresponding to the text feature group “0, 1”.

[0102] As an example, see Figure 6, take a group of video features “6, 7, 8” as an example, and use the video feature group “6, 7, 8” as the query parameter W Q , determine the corresponding key parameter W in the attention mask K Sum parameter W V , perform cross attention calculation.

[0103] Among them, when the video feature group "6, 7, 8" is used as the query parameter W Q When the key parameter W K Sum parameter W V It is composed of the text feature group "0, 1" of the scene corresponding to the video feature group "6, 7, 8" and the video feature groups "6, 7, 8", "9, A, B" and "C, D, E" of all scenes.

[0104] Through the above exemplary embodiment, a complete self-attention calculation is replaced by 2n (n is the number of scenes) cross-attention calculations, eliminating the need for a complete attention mask and reducing redundant calculations between parameters, thereby significantly improving memory usage efficiency and computing efficiency.

[0105] Step S240: taking the feature in the weighted feature corresponding to the original noise information as predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain denoised features.

[0106] In this exemplary embodiment, after obtaining the denoising feature, the method further includes:

[0107] The denoised features are input into the attention layer, and the cascade features are processed with attention masks based on the attention layer to obtain weighted features;

[0108] The feature corresponding to the original noise information in the weighted feature is used as the predicted noise information, and the input denoising feature is denoised based on the predicted noise information to obtain an updated denoising feature;

[0109] The updated denoising features are input into the attention layer, and the cascade features are processed with attention masks based on the attention layer to obtain weighted features.

[0110] …

[0111] The above steps are iteratively performed up to a preset number of times, for example, 50 times, to obtain a final denoising feature, and step S250 is performed based on the final denoising feature.

[0112] Step S250: Decode the denoising feature to obtain video information containing a first number of scenes.

[0113] In this exemplary embodiment, decoding the denoising feature to obtain video information containing a first number of scenes includes:

[0114] Decoding the denoising feature to obtain multiple image frame information;

[0115] Arrange the multiple image frame information in time sequence to obtain video information containing a first number of scenes.

[0116] In a specific implementation, the denoised features (in the form of data, noise) are reconstructed into an image through a decoder (such as a 3D Causal variational self-decoder, etc.).

[0117] Through the above exemplary embodiments, a multi-scene video is generated based on the text containing multiple scenes input by the user. Specifically, video information containing a preset number of scenes is generated based on the attention mask, and video clips of different scenes are generated at the same time, thereby improving the naturalness of the transition between scenes.

[0118] In the above exemplary embodiment, an application scenario of generating a multi-scene video based on a text containing multiple scenes input by a user is introduced. Next, another application scenario is introduced, that is, based on a video containing multiple scenes input by a user and a text for expansion, the input video is expanded to obtain a video containing more scenes, specifically:

[0119] refer to Figure 3 , which is a schematic diagram of an application scenario of the multi-scenario video generation method provided by an exemplary embodiment of the present disclosure.

[0120] The multi-scene video generation method comprises the following steps:

[0121] Step S310: determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result.

[0122] In this disclosure, the analysis results include at least the following two situations:

[0123] (1) The analysis result is that the prompt information is text information containing a first number of scenes, and optionally, the first number is greater than or equal to three.

[0124] (2) The analysis result is that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0125] The present disclosure provides Figure 2 The embodiment shown (i.e., the previous embodiment) is directed to situation (1), as Figure 3 The illustrated embodiment (ie, this embodiment) is directed to situation (2).

[0126] The present disclosure provides a graphical user interface for users to input text information and video information.

[0127] During specific implementation, the user may also input text information through voice input.

[0128] Step S320, in response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, encode the text information to obtain a second number of groups of text features, encode the video information to obtain a third number of groups of video features, and cascade the second number of groups of text features, the third number of groups of video features, and a group of original noise information in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0129] In this exemplary embodiment, the second number is greater than or equal to three, that is, the text information describes contents about at least three scenes.

[0130] The inventors of the present disclosure have found that when the number of scenarios becomes too large, the time and resource consumption required for training and reasoning become unacceptable. To further expand the applicability of the model, in this exemplary embodiment, a conditional mask at the segment level is provided to give the model the ability to autoregressively expand scenarios.

[0131] In the specific implementation, the video clip input by the user is not denoised, and only the original noise information is used to replace the context information of the last video clip to be generated. Therefore, when calculating the loss, only the last video clip to be generated is considered, thereby reducing the time and resource consumption required for training and inference, and realizing the expansion of the input video containing multiple scenes based on the video input by the user and the text used for expansion to obtain a video containing more scenes.

[0132] Step S330: input the cascade features into the attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features.

[0133] The specific implementation of step S330 is similar to that of step S230, so it will not be repeated here.

[0134] Step S340: taking the feature in the weighted feature corresponding to the original noise information as predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain denoised features.

[0135] The specific implementation of step S340 is similar to that of step S240, so it will not be repeated here.

[0136] Step S350: Decode the denoising feature to obtain video information containing a scene.

[0137] The specific implementation of step S350 is similar to that of step S250, so it will not be repeated here.

[0138] In the above exemplary embodiments, two application scenarios of the present disclosure are introduced. The following will introduce the training method of the attention layer used therein, specifically:

[0139] In this exemplary embodiment, the training method of the attention layer includes:

[0140] Get the attention layer to be trained;

[0141] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0142] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0143] In specific implementation, the training process mainly includes two stages: pre-training and supervised fine-tuning.

[0144] In this exemplary embodiment, the pre-training the attention layer to be trained to obtain a pre-trained attention layer includes:

[0145] Determine a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and there is no contextual relationship between the video information;

[0146] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and cascading the fourth group of text features and the fourth group of noisy video features in chronological order to obtain cascade features;

[0147] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0148] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information, and the loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0149] In the specific implementation, in the pre-training stage, n single-scene videos without contextual relationships are spliced ​​into a long video as a training sample. Only the attention mask is introduced for conventional diffusion denoising training without using the conditional mask at the segment level, aiming to gradually adapt the model to the multi-scene generation setting with more frames.

[0150] In specific implementation, the construction method of pre-training data includes: in order to simulate the setting of multi-scene videos, 3 videos are randomly selected from the single-scene videos for splicing. Although these spliced ​​videos have no contextual association, they can help the model gradually adapt to the multi-scene generation task. This splicing method can effectively reduce the dependence on actual multi-scene videos and provide a large number of samples for subsequent model training.

[0151] In this exemplary embodiment, the performing supervised training on the pre-trained attention layer to obtain the attention layer includes:

[0152] Determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and video clips of different scenes have a contextual relationship;

[0153] Encoding the text information to obtain a fifth number group of text features, and encoding the video clip to obtain a fifth number group of video features;

[0154] Performing noise processing on the last group of video features in chronological order based on the original noise information with a preset first probability to obtain a group of noisy video features, cascading the fifth number group of text features, the fifth number minus one group of video features and a group of noisy video features in chronological order to obtain cascade features, inputting the cascade features into the pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, using the features in the weighted features corresponding to the last group of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0155] Performing noise processing on all video features based on the original noise information with a preset second probability to obtain a fifth number group of noisy video features, cascading the fifth number group of text features and the fifth number group of video features in chronological order to obtain cascade features, inputting the cascade features into a pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information in the weighted features to obtain predicted noise information, performing a second loss calculation based on the predicted noise information and the original noise information, and the sum of the first probability and the second probability is one;

[0156] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0157] In the specific implementation, videos of n consecutive scenes with contextual relationships are used as training samples for supervised fine-tuning. This stage aims to enhance the model's ability to generate videos with higher visual coherence between scenes and realize autoregressive expansion of scenes. To this end, a conditional mask at the segment level is introduced, and scene expansion training is performed with a certain probability p, that is, only the video features corresponding to the first n-1 video segments are added with noise, which serves as the context information of the last video segment. During training, only the loss of the last video segment is calculated, instead of calculating the loss of the video features of all video segments as in the pre-training stage. The remaining probability 1-p is supervised using the loss of the pre-training stage.

[0158] In specific implementation, the construction method of fine-tuning data includes: the fine-tuning data is carefully designed for the contextual consistency of multi-scene videos. First, a longer video clip is extracted from the long video dataset, and it is divided into independent shots using a video segmentation tool. The semantic similarity between the shots is calculated through a video-text contrast learning model, and only highly relevant clip combinations are retained. After that, a multimodal large language model is used to generate a consistent description containing role and background information for each clip to ensure the logical coherence of the video in terms of role characteristics and scene background. Finally, these data are used to fine-tune the model to improve the model's cross-scene consistency and semantic alignment capabilities.

[0159] As an example, see Figure 4 During training, scene 1 video information, scene 2 video information and scene 3 video information are input, and the scene 1 text information corresponding to the scene 1 video information is "a person with brown hair and freckles", the scene 2 text information corresponding to the scene 2 video information is "a person with brown hair and a bun", and the scene 3 text information corresponding to the scene 3 video information is "an elderly person wearing an exquisite black dress".

[0160] The above two-stage training process significantly reduces the need for a video collection containing n consecutive scenes with contextual relationships during the entire training process, thereby reducing the manpower required for data collection.

[0161] In this exemplary embodiment, the method further includes:

[0162] Get video information of multiple scenes;

[0163] For video information of different scenes, corresponding text description information is generated based on large language models;

[0164] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0165] In the specific implementation, in order to evaluate the generation effect, a test set containing multiple (for example, 50) multi-scene videos was constructed. Each video scene is driven by a text description generated by a large language model, which covers different backgrounds and roles, emphasizing the diversity and complexity of the scene. The test set also contains predefined role and background information for comparing the visual consistency, semantic alignment, and overall sequence coherence of the generated videos. This test data helps verify the effect and advantages in multi-scene generation tasks.

[0166] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0167] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a multi-scene video generation device.

[0169] refer to Figure 7 , which is a structural diagram of a multi-scene video generation device provided by an exemplary embodiment of the present disclosure.

[0170] The multi-scene video generation device includes the following modules:

[0171] The prompt information analysis module 910 is configured to determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0172] The encoding and noise adding module 920 is configured to, in response to the analysis result that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and cascade the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features;

[0173] The attention mask module 930 is configured to input the cascade features into the attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0174] The denoising and restoration module 940 is configured to use the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0175] The decoding module 950 is configured to decode the denoising feature to obtain video information containing a first number of scenes.

[0176] In some exemplary embodiments, the attention mask module 930 is specifically configured to:

[0177] Dividing the cascade features into the first number group of text features and the first number group of original noise information;

[0178] The text feature and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0179] In some exemplary embodiments, the coding noise adding module 920 is further configured to:

[0180] In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, the video information is encoded to obtain a third number of groups of video features, and the second number of groups of text features, the third number of groups of video features and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0181] In some exemplary embodiments, the coding noise adding module 920 is further configured to:

[0182] Expand the text information based on a large language model to obtain expanded text information;

[0183] The expanded text information is encoded.

[0184] In some exemplary embodiments, the decoding module 950 is configured to:

[0185] Decoding the denoising feature to obtain multiple image frame information;

[0186] Arrange the multiple image frame information in time sequence to obtain video information containing a first number of scenes.

[0187] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0188] Get the attention layer to be trained;

[0189] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0190] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0191] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0192] Determine a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and there is no contextual relationship between the video information;

[0193] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and cascading the fourth group of text features and the fourth group of noisy video features in chronological order to obtain cascade features;

[0194] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0195] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information, and the loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0196] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0197] Determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and video clips of different scenes have a contextual relationship;

[0198] Encoding the text information to obtain a fifth number of groups of text features, and encoding the video clips to obtain a fifth number of groups of video features;

[0199] Performing noise processing on the last group of video features in chronological order based on the original noise information with a preset first probability to obtain a group of noisy video features, cascading the fifth number group of text features, the fifth number minus one group of video features and a group of noisy video features in chronological order to obtain cascade features, inputting the cascade features into the pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, using the features in the weighted features corresponding to the last group of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0200] Performing noise processing on all video features based on the original noise information with a preset second probability to obtain a fifth number group of noisy video features, cascading the fifth number group of text features and the fifth number group of video features in chronological order to obtain cascade features, inputting the cascade features into a pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information in the weighted features to obtain predicted noise information, performing a second loss calculation based on the predicted noise information and the original noise information, and the sum of the first probability and the second probability is one;

[0201] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0202] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0203] Get video information of multiple scenes;

[0204] For video information of different scenes, corresponding text description information is generated based on large language models;

[0205] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0206] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0207] The device of the above embodiment is used to implement the corresponding multi-scene video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0208] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-scene video generation method described in any of the above embodiments is implemented.

[0209] Figure 8 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0210] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0211] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0212] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0213] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0214] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0215] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0216] The electronic device of the above embodiment is used to implement the corresponding multi-scene video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0217] The memory 1020 stores machine-readable instructions executable by the processor 1010. When the electronic device is running, the processor 1010 communicates with the memory 1020 via the bus 1030, so that the processor 1010 executes the following instructions when running:

[0218] Determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0219] In response to the analysis result that the prompt information is text information containing a first number of scenes, the text information is encoded to obtain a first number of groups of text features, and the first number of groups of text features are cascaded with the first number of groups of original noise information in chronological order to obtain cascade features;

[0220] Inputting the cascade features into an attention layer, and performing attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0221] Taking the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0222] The denoising feature is decoded to obtain video information including a first number of scenes.

[0223] In a possible implementation, in the instruction executed by the processor 1010, performing attention mask processing on the cascade feature based on the attention layer includes:

[0224] Dividing the cascade features into the first number group of text features and the first number group of original noise information;

[0225] The text feature and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0226] In a possible implementation manner, in the instruction executed by the processor 1010, the method further includes:

[0227] In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, the video information is encoded to obtain a third number of groups of video features, and the second number of groups of text features, the third number of groups of video features and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0228] In a possible implementation manner, in the instruction executed by the processor 1010, before encoding the text information, the method further includes:

[0229] Expand the text information based on a large language model to obtain expanded text information;

[0230] Then, encoding the text information includes:

[0231] The expanded text information is encoded.

[0232] In a possible implementation manner, in the instructions executed by the processor 1010, decoding the denoising feature to obtain video information including a first number of scenes includes:

[0233] Decoding the denoising feature to obtain multiple image frame information;

[0234] Arrange the multiple image frame information in time sequence to obtain video information containing a first number of scenes.

[0235] In a possible implementation, in the instruction executed by the processor 1010, the training method of the attention layer includes:

[0236] Get the attention layer to be trained;

[0237] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0238] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0239] In a possible implementation, in the instructions executed by the processor 1010, pre-training the attention layer to be trained to obtain a pre-trained attention layer includes:

[0240] Determine a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and there is no contextual relationship between the video information;

[0241] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and cascading the fourth group of text features and the fourth group of noisy video features in chronological order to obtain cascade features;

[0242] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0243] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information, and the loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0244] In a possible implementation, in the instructions executed by the processor 1010, the performing supervised training on the pre-trained attention layer to obtain the attention layer includes:

[0245] Determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and video clips of different scenes have a contextual relationship;

[0246] Encoding the text information to obtain a fifth number of groups of text features, and encoding the video clips to obtain a fifth number of groups of video features;

[0247] Performing noise processing on the last group of video features in chronological order based on the original noise information with a preset first probability to obtain a group of noisy video features, cascading the fifth number group of text features, the fifth number minus one group of video features and a group of noisy video features in chronological order to obtain cascade features, inputting the cascade features into the pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, using the features in the weighted features corresponding to the last group of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0248] Performing noise processing on all video features based on the original noise information with a preset second probability to obtain a fifth number group of noisy video features, cascading the fifth number group of text features and the fifth number group of video features in chronological order to obtain cascade features, inputting the cascade features into a pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information in the weighted features to obtain predicted noise information, performing a second loss calculation based on the predicted noise information and the original noise information, and the sum of the first probability and the second probability is one;

[0249] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0250] In a possible implementation manner, in the instruction executed by the processor 1010, the method further includes:

[0251] Get video information of multiple scenes;

[0252] For video information of different scenes, corresponding text description information is generated based on large language models;

[0253] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0254] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-scene video generation method described in any of the above embodiments.

[0255] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0256] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0257] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multi-scene video generation method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0258] Based on the same inventive concept, corresponding to the multi-scene video generation method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the multi-scene video generation method. Corresponding to the execution subject corresponding to each step in each embodiment of the multi-scene video generation method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0259] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the multi-scene video generation method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0260] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0261] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0262] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0263] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0264] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0265] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0266] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0267] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0268] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0269] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0270] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0271] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0272] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0273] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0274] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0275] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A multi-scene video generation method, characterized in that: include: Determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result; In response to the analysis result that the prompt information is text information containing a first number of scenes, the text information is encoded to obtain a first number of groups of text features, and the first number of groups of text features are cascaded with the first number of groups of original noise information in chronological order to obtain cascade features; Inputting the cascade features into an attention layer, and performing attention mask processing on the cascade features based on the attention layer to obtain weighted features; Taking the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature; The denoising feature is decoded to obtain video information including a first number of scenes.

2. The method according to claim 1, characterized in that The performing attention mask processing on the cascade features based on the attention layer includes: Dividing the cascade features into the first number group of text features and the first number group of original noise information; The text feature and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

3. The method according to claim 1, characterized in that The method further comprises: In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, the video information is encoded to obtain a third number of groups of video features, and the second number of groups of text features, the third number of groups of video features and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

4. The method according to claim 1, characterized in that: Before encoding the text information, the method further includes: Expand the text information based on a large language model to obtain expanded text information; Then, encoding the text information includes: The expanded text information is encoded.

5. The method according to claim 1, characterized in that The decoding of the denoising feature to obtain video information containing a first number of scenes includes: Decoding the denoising feature to obtain multiple image frame information; Arrange the multiple image frame information in time sequence to obtain video information containing a first number of scenes.

6. The method according to claim 1, characterized in that The training method of the attention layer includes: Get the attention layer to be trained; Pre-training the attention layer to be trained to obtain a pre-trained attention layer; The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

7. The method according to claim 6, characterized in that The pre-training of the attention layer to be trained to obtain a pre-trained attention layer includes: Determine a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and there is no contextual relationship between the video information; Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and cascading the fourth group of text features and the fourth group of noisy video features in chronological order to obtain cascade features; The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features; The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information, and the loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

8. The method according to claim 6, characterized in that The performing supervised training on the pre-trained attention layer to obtain the attention layer includes: Determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and video clips of different scenes have a contextual relationship; Encoding the text information to obtain a fifth number group of text features, and encoding the video clip to obtain a fifth number group of video features; Performing noise processing on the last group of video features in chronological order based on the original noise information with a preset first probability to obtain a group of noisy video features, cascading the fifth number group of text features, the fifth number minus one group of video features and a group of noisy video features in chronological order to obtain cascade features, inputting the cascade features into the pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, using the features in the weighted features corresponding to the last group of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information; Performing noise processing on all video features based on the original noise information with a preset second probability to obtain a fifth number group of noisy video features, cascading the fifth number group of text features and the fifth number group of video features in chronological order to obtain cascade features, inputting the cascade features into a pre-trained attention layer, performing attention mask processing on the cascade features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information in the weighted features to obtain predicted noise information, performing a second loss calculation based on the predicted noise information and the original noise information, and the sum of the first probability and the second probability is one; Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

9. The method according to claim 6, characterized in that The method further comprises: Get video information of multiple scenes; For video information of different scenes, corresponding text description information is generated based on large language models; The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

10. A multi-scene video generation device, characterized in that: include: A prompt information analysis module is configured to determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result; The encoding and noise adding module is configured to, in response to the analysis result that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and cascade the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features; An attention mask module is configured to input the cascade features into an attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features; a denoising and restoration module, configured to use the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature; The decoding module is configured to decode the denoising feature to obtain video information containing a first number of scenes.

11. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.

Images