Multi-scene video generation method and device, electronic equipment, storage medium and program product

Abstract

The present disclosure provides a multi-scene video generation method and device, electronic equipment, storage medium and program product, the method comprising: when the prompt information input by the user is text information containing a first number of scenes, encoding the text information to obtain a first number of groups of text features, and concatenating the first number of groups of original noise information according to the time sequence to obtain concatenated features; inputting the concatenated features into an attention layer, and performing attention mask processing on the concatenated features to obtain weighted features; taking the features corresponding to the original noise information in the weighted features as predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain denoised features; and decoding the denoised features to obtain video information containing the first number of scenes. The present disclosure generates video information containing a preset number of scenes based on attention mask, and simultaneously generates video clips of different scenes, thereby improving the natural degree of transition between scenes.

Description

Technical Field

[0001] The present disclosure relates to the field of video generation technology, and in particular to a multi-scene video generation method, device, electronic device, storage medium, and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present disclosure that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Multi-scene video refers to a video that contains multiple different scenes, and these scenes may be switched in time or space.

[0004] Multi-scene video generation refers to generating videos from multi-scene descriptions while maintaining semantic consistency and temporal order among scenes.

[0005] When generating a multi-scene video, the related technology usually first generates multiple single-scene videos, and then splices the multiple single-scene videos together to obtain a multi-scene video.

[0006] However, this approach often leaves noticeable seams between scenes, resulting in unnatural transitions between scenes in the video. Summary of the Invention

[0007] In view of this, the purpose of the present disclosure is to propose a multi-scene video generation method, device, electronic device, storage medium and program product, which at least to a certain extent solve one of the technical problems in the related art.

[0008] Based on the above objectives, the first aspect of the exemplary embodiments of the present disclosure provides a multi-scene video generation method, including:

[0009] Determine prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0010] In response to the analysis result that the prompt information is text information including a first number of scenes, encoding the text information to obtain a first number of groups of text features, and cascading the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features;

[0011] Inputting the cascade features into the attention layer, performing attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0012] using the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0013] The denoising features are decoded to obtain video information including a first number of scenes.

[0014] Based on the same inventive concept, a second aspect of the exemplary embodiments of the present disclosure provides a multi-scene video generation device, including:

[0015] a prompt information analysis module configured to determine prompt information input by a user, analyze the prompt information, and obtain an analysis result;

[0016] an encoding and noise adding module, configured to, in response to the analysis result indicating that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and concatenate the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain concatenated features;

[0017] an attention mask module, configured to input the cascade features into an attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0018] a denoising and restoration module configured to use a feature in the weighted feature corresponding to the original noise information as predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0019] The decoding module is configured to decode the denoising feature to obtain video information containing a first number of scenes.

[0020] Based on the same inventive concept, the third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0021] Based on the same inventive concept, a fourth aspect of the exemplary embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0022] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer is caused to execute the method described in the first aspect.

[0023] As can be seen from the above, the multi-scene video generation method, device, electronic device, storage medium and program product provided by the embodiments of the present disclosure include: determining the prompt information input by the user, analyzing the prompt information to obtain an analysis result; in response to the analysis result that the prompt information is text information containing a first number of scenes, encoding the text information to obtain a first number of groups of text features, cascading the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features; inputting the cascade features into the attention layer, performing attention masking on the cascade features based on the attention layer to obtain weighted features; using the features in the weighted features corresponding to the original noise information as predicted noise information, denoising the original noise information based on the predicted noise information to obtain denoised features; decoding the denoised features to obtain video information containing the first number of scenes. The present disclosure generates video information containing a preset number of scenes based on the attention mask. Since video clips of different scenes are generated simultaneously, no obvious seams are left between the scenes, thereby improving the naturalness of the transition between scenes in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an application scenario of the multi-scene video generation method provided by an exemplary embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram of a flow chart of a multi-scene video generation method provided by an exemplary embodiment of the present disclosure;

[0027] Figure 3 Another flowchart of a multi-scene video generation method provided by an exemplary embodiment of the present disclosure;

[0028] Figure 4 A schematic diagram of a training method for an attention layer provided by an exemplary embodiment of the present disclosure;

[0029] Figure 5 A schematic diagram of a group attention mechanism provided for an exemplary embodiment of the present disclosure;

[0030] Figure 6 Another schematic diagram of a group attention mechanism provided for an exemplary embodiment of the present disclosure;

[0031] Figure 7 A schematic structural diagram of a multi-scene video generation device provided by an exemplary embodiment of the present disclosure;

[0032] Figure 8 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0034] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. Thus, based on the prompt message, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the technical solution of this application.

[0035] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0036] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0037] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0038] To make the objectives, technical solutions, and advantages of the present disclosure more clearly understood, the principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0039] It should be understood herein that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0040] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "an" before an element does not exclude the presence of multiple such elements.

[0041] The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0042] As described in the background art, multi-scene video refers to a video that contains multiple different scenes, and these scenes may be switched in time or space.

[0043] Multi-scene video generation refers to generating videos from multi-scene descriptions while maintaining semantic consistency and temporal order among scenes.

[0044] However, the inventors of the present disclosure have discovered that connecting these keyframes often results in unnatural motion transitions because the keyframe sampling process fails to fully consider the temporal positioning and related motion dynamics in the multi-scene video timeline.

[0045] In order to solve the above problems, the present disclosure provides a multi-scene video generation solution, which specifically includes: when the prompt information input by the user is text information containing a first number of scenes, encoding the text information to obtain a first number of groups of text features, and cascading them with the first number of groups of original noise information in chronological order to obtain cascade features; inputting the cascade features into the attention layer, performing attention masking on the cascade features to obtain weighted features; using the features in the weighted features corresponding to the original noise information as predicted noise information, and denoising the original noise information based on the predicted noise information to obtain denoised features; decoding the denoised features to obtain video information containing the first number of scenes.

[0046] The present disclosure generates video information containing a preset number of scenes based on an attention mask, and simultaneously generates video clips of different scenes, thereby improving the naturalness of transitions between scenes.

[0047] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0048] refer to Figure 1 , which is a schematic diagram of an application scenario of the multi-scene video generation method provided by an exemplary embodiment of the present disclosure.

[0049] This application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, the server 102, and the data storage system 103 can be connected via a wired or wireless communication network to achieve data interaction.

[0050] The terminal device 101 can be an electronic device close to the user with data transmission and multimedia input / output functions, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA), or other electronic devices capable of implementing the above functions. The electronic device may include a processor and a display screen with touch input function, the display screen is used to present a graphical user interface, and the graphical user interface can display an application interface. The processor is used to process application data, generate the graphical user interface, and control the display of the graphical user interface on the display screen.

[0051] Both server 102 and data storage system 103 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0052] In some exemplary embodiments, the multi-scene video generation method may be run on the terminal device 101 or the server 102 .

[0053] When the multi-scene video generation method runs on the server 102, the server 102 is used to provide a multi-scene video generation service to the user of the terminal device 101. The terminal device 101 is installed with a client that communicates with the server 102. The user can input prompt information through the client, and the client sends the prompt information to the server 102. The server 102 determines the prompt information input by the user, analyzes the prompt information, and obtains the analysis result.

[0054] When the analysis result is a prompt message containing text information of the first number of scenes:

[0055] The server 102 encodes the text information to obtain a first number of groups of text features, concatenates the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain concatenated features; inputs the concatenated features into the attention layer, performs attention masking on the concatenated features based on the attention layer to obtain weighted features; uses the features in the weighted features corresponding to the original noise information as predicted noise information, denoises the original noise information based on the predicted noise information to obtain denoised features; and decodes the denoised features to obtain video information containing a first number of scenes.

[0056] When the analysis result is that the prompt information is text information containing the second number of scenes and video information containing the third number of scenes:

[0057] Server 102 encodes the text information to obtain a second set of text features, encodes the video information to obtain a third set of video features, and concatenates the second set of text features, the third set of video features, and a set of original noise information in chronological order to obtain concatenated features, wherein the second number is greater than or equal to three, and the third number is the second number minus one; inputs the concatenated features into the attention layer, performs attention masking on the concatenated features based on the attention layer to obtain weighted features; uses the features in the weighted features corresponding to the original noise information as predicted noise information, and denoises the original noise information based on the predicted noise information to obtain denoised features; decodes the denoised features to obtain video information containing a scene.

[0058] After generating video information containing a scene, the server 102 can send the video information containing a scene to the client, and the client displays the video information containing a scene to the user; the server 102 can also splice the video information containing a scene after the video information containing a third number of scenes input by the user and send it to the client, and the client displays the spliced ​​video information to the user.

[0059] The data storage system 103 stores a large amount of training data. Server 102 can train the attention layer based on this large amount of training data, enabling the attention layer to predict input features based on attention mask processing. The sources of training data include but are not limited to existing databases, data crawled from the internet, or data uploaded by users when using the client. When the accuracy of the attention layer output meets certain requirements, server 102 can provide users with multi-scene video generation services based on the attention layer. At the same time, server 102 can also continuously optimize the attention layer based on the newly added training data.

[0060] The following combination Figure 1The present invention will use the application scenario of the present invention to describe the training method of the attention layer and the multi-scene video generation method according to the exemplary embodiment of the present invention. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not limited in this respect. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0061] The present disclosure has at least two application scenarios: one is to generate a multi-scene video based on a text input by a user containing multiple scenes; the other is to expand the input video based on a video containing multiple scenes and text for expansion input by a user to obtain a video containing more scenes. The following first introduces the application scenario of generating a multi-scene video based on a text input by a user containing multiple scenes:

[0062] refer to Figure 2 , which is a schematic diagram of an application scenario of the multi-scene video generation method provided by an exemplary embodiment of the present disclosure.

[0063] The multi-scene video generation method includes the following steps:

[0064] Step S210: Determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result.

[0065] In this disclosure, the analysis results include at least the following two situations:

[0066] (1) The analysis result is that the prompt information is text information containing a first number of scenes, and optionally, the first number is greater than or equal to three.

[0067] (2) The analysis result is that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0068] The present disclosure provides Figure 2 The embodiment shown (i.e., this embodiment) is directed to case (1), where Figure 3 The embodiment shown (ie, the next embodiment) is directed to situation (2).

[0069] The present disclosure provides a graphical user interface for users to input text information and video information.

[0070] During specific implementation, users can also input text information through voice input.

[0071] Step S220: In response to the analysis result that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and cascade the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features.

[0072] In this exemplary embodiment, the first number is greater than or equal to three, that is, the text information describes contents related to at least three scenes.

[0073] In this exemplary embodiment, in order to improve the richness of the generated multi-scene video, before encoding the text information, the method further includes:

[0074] Expand the text information based on a large language model to obtain expanded text information;

[0075] Then, encoding the text information includes:

[0076] The expanded text information is encoded.

[0077] During specific implementation, each scene in the text information containing the first number of scenes is expanded respectively.

[0078] It should be noted that the expanded text information still only includes the first number of scenes, and the number of scenes is not expanded.

[0079] In this exemplary embodiment, text information is encoded to obtain a first number of groups of text features, wherein each group of text features corresponds to a scene.

[0080] In this exemplary embodiment, the original noise information is random noise.

[0081] It should be noted that the number of groups of original noise information is the same as the number of scenes, and is used to finally generate video clips having the same number of scenes.

[0082] In this exemplary embodiment, the first number of groups of text features and the first number of groups of original noise information are concatenated in chronological order to obtain concatenated features, where:

[0083] The first number of groups of text features are located in front, and the first number of groups of original noise information are located in the back.

[0084] Step S230: input the cascade features into the attention layer, perform attention mask processing on the cascade features based on the attention layer, and obtain weighted features.

[0085] Among them, considering that directly predicting based on cascade features will destroy the fine-grained alignment relationship between the original scene and its corresponding text annotation, the present disclosure introduces an attention mask to establish such a fine-grained one-to-one alignment, so that each scene segment can focus on its corresponding text annotation more effectively.

[0086] In this exemplary embodiment, attention mask processing is performed on the cascade features based on the attention layer.

[0087] In specific implementation, the attention layer includes a 3D full attention layer, the core of which is the self-attention mechanism, which allows the 3D full attention layer to dynamically focus on different parts of the input features when processing data.

[0088] The attention mechanism is controlled by creating a sparse mask so that the 3D full attention layer only focuses on specific parts of the input features.

[0089] By defining a mask function, the function returns a Boolean mask based on the input query and the index position of the key, indicating which keys at which positions the query should focus on. For example, when the query is a text feature, each query is set to focus only on the text features and video features belonging to the same scene (when no video features exist, they are replaced with original noise information); when the query is a video feature (when no video features exist, they are replaced with original noise information), each query is set to focus only on the text features and all video features belonging to the same scene (when no video features exist, they are replaced with original noise information).

[0090] As an example, see Figure 4 , where "0, 1", "2, 3", and "4, 5" are three groups of text features, and "6, 7, 8", "9, A, B", and "C, D, E" are three groups of video features (when there is no video feature, the original noise information is used instead). Among them, "0, 1" is the text description of "6, 7, 8", "2, 3" is the text description of "9, A, B", and "4, 5" is the text description of "C, D, E".

[0091] When the mask is not used, the prediction of “6, 7, 8” will not only focus on its text description “0, 1”, but also on the text descriptions “2, 3” and “4, 5”, which will obviously affect the accuracy of the generated video clip.

[0092] Through Figure 4 The symmetrical binary attention mask shown in only focuses on the text descriptions “0, 1” when predicting “6, 7, 8”, and ignores the text descriptions “2, 3” and “4, 5”, which improves the accuracy of the generated video clips.

[0093] Through the above exemplary embodiments, each scene segment can be more effectively focused on its corresponding text annotation, thereby improving the accuracy of the generated video segment.

[0094] However, the inventors of the present disclosure have found that in the above exemplary embodiment, due to the long total length of the sequence of cascaded features, the memory usage of the attention mask is significantly increased. To solve this problem, the present disclosure also provides a grouped attention mechanism, specifically:

[0095] In this exemplary embodiment, performing attention mask processing on the cascade features based on the attention layer includes:

[0096] Dividing the concatenated features into the first number of groups of text features and the first number of groups of original noise information;

[0097] The text features and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0098] When implementing it specifically, Figure 4 The symmetrical binary attention mask shown in the figure is used to illustrate that the entire one-dimensional cascade feature sequence "0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E" is divided into 2n (n=3) groups, including 3 text feature groups "0, 1", "2, 3", "4, 5" and 3 video feature groups (when there is no video feature, it is replaced by the original noise information) "6, 7, 8", "9, A, B", "C, D, E".

[0099] Each set of features is treated as a query parameter, and the corresponding key parameter and value parameter are found in the attention mask, and then a cross attention calculation is performed.

[0100] As an example, see Figure 5 , take a set of text features “0, 1” as an example to illustrate, and use the text feature group “0, 1” as the query parameter W Q , determine the corresponding key parameter W in the attention mask K Sum parameter W V , perform cross attention calculation.

[0101] Among them, when the text feature group "0, 1" is used as the query parameter W Q When the key parameter W K Sum parameter W V It is composed of the text feature group "0, 1" and the video feature group "6, 7, 8" of the scene corresponding to the text feature group "0, 1".

[0102] As an example, see Figure 6, take a set of video features “6, 7, 8” as an example to illustrate, and use the video feature group “6, 7, 8” as the query parameter W Q , determine the corresponding key parameter W in the attention mask K Sum parameter W V , perform cross attention calculation.

[0103] Among them, when the video feature group "6, 7, 8" is used as the query parameter W Q When the key parameter W K Sum parameter W V It is composed of the text feature group "0, 1" of the scene corresponding to the video feature group "6, 7, 8" and the video feature groups "6, 7, 8", "9, A, B" and "C, D, E" of all scenes.

[0104] Through the above exemplary embodiment, a complete self-attention calculation is replaced by 2n (n is the number of scenes) cross-attention calculations, eliminating the need for a complete attention mask and reducing redundant calculations between parameters, thereby significantly improving memory usage efficiency and computing efficiency.

[0105] Step S240: Use the feature corresponding to the original noise information in the weighted feature as predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain denoised features.

[0106] In this exemplary embodiment, after obtaining the denoising features, the method further includes:

[0107] The denoised features are input into the attention layer, and the cascade features are processed with attention mask based on the attention layer to obtain weighted features;

[0108] The feature corresponding to the original noise information in the weighted feature is used as the predicted noise information, and the input denoising feature is denoised based on the predicted noise information to obtain an updated denoising feature;

[0109] The updated denoising features are input into the attention layer, and the cascade features are subjected to attention mask processing based on the attention layer to obtain weighted features;

[0110] …

[0111] The above steps are iterated for a preset number of times, for example, 50 times, to obtain a final denoising feature, and step S250 is performed based on the final denoising feature.

[0112] Step S250: Decode the denoising features to obtain video information containing a first number of scenes.

[0113] In this exemplary embodiment, decoding the denoising feature to obtain video information containing a first number of scenes includes:

[0114] Decoding the denoising features to obtain multiple image frame information;

[0115] Arrange the multiple image frame information in chronological order to obtain video information containing a first number of scenes.

[0116] In a specific implementation, the denoised features (data in the form of noise) are reconstructed into an image through a decoder (such as a 3D Causal variational self-decoder, etc.).

[0117] Through the above exemplary embodiments, a multi-scene video is generated based on the text containing multiple scenes input by the user. Specifically, video information containing a preset number of scenes is generated based on the attention mask, and video clips of different scenes are generated at the same time, thereby improving the naturalness of the transition between scenes.

[0118] In the above exemplary embodiment, an application scenario of generating a multi-scene video based on a text containing multiple scenes input by a user is introduced. Next, another application scenario is introduced, that is, based on a video containing multiple scenes input by a user and text for expansion, the input video is expanded to obtain a video containing more scenes. Specifically:

[0119] refer to Figure 3 , which is a schematic diagram of an application scenario of the multi-scene video generation method provided by an exemplary embodiment of the present disclosure.

[0120] The multi-scene video generation method includes the following steps:

[0121] Step S310: Determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result.

[0122] In this disclosure, the analysis results include at least the following two situations:

[0123] (1) The analysis result is that the prompt information is text information containing a first number of scenes, and optionally, the first number is greater than or equal to three.

[0124] (2) The analysis result is that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0125] The present disclosure provides Figure 2 The embodiment shown (i.e., the previous embodiment) is directed to situation (1), as Figure 3 The embodiment shown (ie, this embodiment) is directed to situation (2).

[0126] The present disclosure provides a graphical user interface for users to input text information and video information.

[0127] During specific implementation, users can also input text information through voice input.

[0128] Step S320: In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, the video information is encoded to obtain a third number of groups of video features, and the second number of groups of text features, the third number of groups of video features, and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0129] In this exemplary embodiment, the second number is greater than or equal to three, that is, the text information describes contents related to at least three scenes.

[0130] The inventors of this disclosure have found that when the number of scenarios becomes too large, the time and resource consumption required for training and inference become unacceptable. To further expand the applicability of the model, this exemplary embodiment provides a segment-level conditional mask to give the model the ability to autoregressively expand scenarios.

[0131] In specific implementation, the video clip input by the user is not denoised. As the context information of the last video clip to be generated, only the original noise information is used to replace the last video clip to be generated. Therefore, when calculating the loss, only the last video clip to be generated is considered, thereby reducing the time and resource consumption required for training and inference, and realizing the expansion of the input video based on the video containing multiple scenes input by the user and the text used for expansion to obtain a video containing more scenes.

[0132] Step S330: input the cascade features into the attention layer, perform attention mask processing on the cascade features based on the attention layer, and obtain weighted features.

[0133] The specific implementation of step S330 is similar to that of step S230, so it will not be repeated here.

[0134] Step S340: Use the feature corresponding to the original noise information in the weighted feature as predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain denoised features.

[0135] The specific implementation of step S340 is similar to that of step S240, so it will not be repeated here.

[0136] Step S350: Decode the denoising features to obtain video information containing a scene.

[0137] The specific implementation of step S350 is similar to that of step S250, so it will not be repeated here.

[0138] In the above exemplary embodiments, two application scenarios of the present disclosure are introduced. The following describes the training method of the attention layer used therein, specifically:

[0139] In this exemplary embodiment, the training method of the attention layer includes:

[0140] Get the attention layer to be trained;

[0141] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0142] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0143] In specific implementation, the training process mainly includes two stages: pre-training and supervised fine-tuning.

[0144] In this exemplary embodiment, pre-training the attention layer to be trained to obtain a pre-trained attention layer includes:

[0145] Determining a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and the video information has no contextual relationship;

[0146] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and concatenating the fourth group of text features and the fourth group of noisy video features in chronological order to obtain a cascade feature;

[0147] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0148] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information. The loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0149] In practice, during the pre-training phase, n context-free, single-scene videos are concatenated into a single long video as a training sample. Conventional diffusion denoising training is performed using only attention masks, without using segment-level conditional masks. This allows the model to gradually adapt to multi-scene generation settings with more frames.

[0150] In practice, pre-training data is constructed by randomly selecting three single-scene videos and splicing them together to simulate a multi-scene video setup. While these spliced ​​videos lack context, they help the model gradually adapt to multi-scene generation tasks. This splicing method effectively reduces reliance on actual multi-scene videos while providing a large number of samples for subsequent model training.

[0151] In this exemplary embodiment, the performing supervised training on the pre-trained attention layer to obtain the attention layer includes:

[0152] determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and the video clips of different scenes have a contextual relationship;

[0153] Encoding the text information to obtain a fifth set of text features, and encoding the video clip to obtain a fifth set of video features;

[0154] Noising the last set of video features in chronological order based on the original noise information with a preset first probability to obtain a set of noisy video features, concatenating the fifth set of text features, the fifth set of video features minus one set, and the set of noisy video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, using features in the weighted features corresponding to the last set of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0155] Noising all video features based on the original noise information with a preset second probability to obtain a fifth group of noisy video features, concatenating the fifth group of text features and the fifth group of video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information from the weighted features to obtain predicted noise information, and performing a second loss calculation based on the predicted noise information and the original noise information, where the sum of the first probability and the second probability is one;

[0156] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0157] In practice, supervised fine-tuning is performed using videos of n consecutive scenes with contextual relationships as training samples. This stage aims to enhance the model's ability to generate videos with greater visual coherence between scenes and enable autoregressive scene expansion. To this end, a clip-level conditional mask is introduced, and scene expansion training is performed with a certain probability p. This means that noise is added only to the video features corresponding to the first n-1 video clips, serving as contextual information for the last video clip. During training, only the loss for the last video clip is calculated, rather than the loss for the video features of all video clips as in the pre-training stage. The remaining probability 1-p is supervised using the loss from the pre-training stage.

[0158] In specific implementation, the fine-tuning data is constructed in the following ways: The fine-tuning data is carefully designed to ensure contextual consistency across multi-scene videos. First, a longer video clip is extracted from a long video dataset and segmented into independent shots using a video segmentation tool. A video-text comparative learning model is then used to calculate the semantic similarity between the shots, retaining only highly relevant clip combinations. A multimodal large language model is then used to generate a consistent description of each clip containing character and background information, ensuring the logical coherence of the video in terms of character traits and scene background. Ultimately, this data is used to fine-tune the model, improving its cross-scene consistency and semantic alignment capabilities.

[0159] As an example, see Figure 4 During training, scene 1 video information, scene 2 video information, and scene 3 video information are input, and the scene 1 text information corresponding to the scene 1 video information is "a person with brown hair and freckles", the scene 2 text information corresponding to the scene 2 video information is "a person with brown hair and a bun", and the scene 3 text information corresponding to the scene 3 video information is "an elderly person wearing an exquisite black dress".

[0160] The above two-stage training process significantly reduces the need for a video collection containing n consecutive scenes with contextual relationships during the entire training process, thereby reducing the manpower required for data collection.

[0161] In this exemplary embodiment, the method further includes:

[0162] Get video information of multiple scenes;

[0163] For video information of different scenes, corresponding text description information is generated based on large language models;

[0164] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0165] In practice, to evaluate the generation results, a test set consisting of multiple (e.g., 50) multi-scene videos was constructed. Each video scene is driven by a text description generated by a large language model. These descriptions cover different backgrounds and characters, emphasizing the diversity and complexity of the scenes. The test set also contains predefined character and background information for comparing the visual consistency, semantic alignment, and overall sequence coherence of the generated videos. This test data helps verify the effectiveness and advantages of the multi-scene generation task.

[0166] It should be noted that the method of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The method of the embodiments of the present disclosure can also be applied in a distributed scenario, where multiple devices cooperate to perform the method. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the method.

[0167] It should be noted that the above description is limited to some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0168] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a multi-scene video generation device.

[0169] refer to Figure 7 , which is a structural diagram of a multi-scene video generation device provided by an exemplary embodiment of the present disclosure.

[0170] The multi-scene video generation device includes the following modules:

[0171] The prompt information analysis module 910 is configured to determine the prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0172] The encoding and noise adding module 920 is configured to, in response to the analysis result indicating that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and concatenate the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain concatenated features;

[0173] an attention mask module 930 configured to input the cascade features into an attention layer, and perform attention masking on the cascade features based on the attention layer to obtain weighted features;

[0174] The denoising and restoration module 940 is configured to use the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0175] The decoding module 950 is configured to decode the denoising feature to obtain video information containing a first number of scenes.

[0176] In some exemplary embodiments, the attention mask module 930 is specifically configured to:

[0177] Dividing the concatenated features into the first number of groups of text features and the first number of groups of original noise information;

[0178] The text features and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0179] In some exemplary embodiments, the coding noise adding module 920 is further configured to:

[0180] In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, and the video information is encoded to obtain a third number of groups of video features. The second number of groups of text features, the third number of groups of video features, and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0181] In some exemplary embodiments, the coding noise adding module 920 is further configured to:

[0182] Expand the text information based on a large language model to obtain expanded text information;

[0183] The expanded text information is encoded.

[0184] In some exemplary embodiments, the decoding module 950 is configured to:

[0185] Decoding the denoising features to obtain multiple image frame information;

[0186] Arrange the multiple image frame information in chronological order to obtain video information containing a first number of scenes.

[0187] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0188] Get the attention layer to be trained;

[0189] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0190] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0191] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0192] Determining a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and the video information has no contextual relationship;

[0193] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and concatenating the fourth group of text features and the fourth group of noisy video features in chronological order to obtain a cascade feature;

[0194] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0195] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information. The loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0196] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0197] determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and the video clips of different scenes have a contextual relationship;

[0198] Encoding the text information to obtain a fifth set of text features, and encoding the video clip to obtain a fifth set of video features;

[0199] Noising the last set of video features in chronological order based on the original noise information with a preset first probability to obtain a set of noisy video features, concatenating the fifth set of text features, the fifth set of video features minus one set, and the set of noisy video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, using features in the weighted features corresponding to the last set of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0200] Noising all video features based on the original noise information with a preset second probability to obtain a fifth group of noisy video features, concatenating the fifth group of text features and the fifth group of video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information from the weighted features to obtain predicted noise information, and performing a second loss calculation based on the predicted noise information and the original noise information, where the sum of the first probability and the second probability is one;

[0201] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0202] In some exemplary embodiments, the attention mask module 930′ is further configured to:

[0203] Get video information of multiple scenes;

[0204] For video information of different scenes, corresponding text description information is generated based on large language models;

[0205] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0206] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0207] The device of the above embodiment is used to implement the corresponding multi-scene video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0208] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the multi-scene video generation method described in any of the above embodiments is implemented.

[0209] Figure 8 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0210] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0211] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0212] The input / output interface 1030 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0213] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0214] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0215] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0216] The electronic device of the above embodiment is used to implement the corresponding multi-scene video generation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0217] The memory 1020 stores machine-readable instructions executable by the processor 1010. When the electronic device is running, the processor 1010 communicates with the memory 1020 via the bus 1030, so that the processor 1010 executes the following instructions during operation:

[0218] Determine prompt information input by the user, analyze the prompt information, and obtain an analysis result;

[0219] In response to the analysis result that the prompt information is text information including a first number of scenes, encoding the text information to obtain a first number of groups of text features, and cascading the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features;

[0220] Inputting the cascade features into the attention layer, performing attention mask processing on the cascade features based on the attention layer to obtain weighted features;

[0221] using the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature;

[0222] The denoising features are decoded to obtain video information including a first number of scenes.

[0223] In one possible implementation, in the instructions executed by the processor 1010, performing attention mask processing on the cascade features based on the attention layer includes:

[0224] Dividing the concatenated features into the first number of groups of text features and the first number of groups of original noise information;

[0225] The text features and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

[0226] In a possible implementation, in the instructions executed by the processor 1010, the method further includes:

[0227] In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, and the video information is encoded to obtain a third number of groups of video features. The second number of groups of text features, the third number of groups of video features, and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

[0228] In a possible implementation, in the instructions executed by the processor 1010, before encoding the text information, the method further includes:

[0229] Expand the text information based on a large language model to obtain expanded text information;

[0230] Then, encoding the text information includes:

[0231] The expanded text information is encoded.

[0232] In one possible implementation, in the instructions executed by the processor 1010, decoding the denoising feature to obtain video information including a first number of scenes includes:

[0233] Decoding the denoising features to obtain multiple image frame information;

[0234] Arrange the multiple image frame information in chronological order to obtain video information containing a first number of scenes.

[0235] In one possible implementation, in the instructions executed by the processor 1010, the training method of the attention layer includes:

[0236] Get the attention layer to be trained;

[0237] Pre-training the attention layer to be trained to obtain a pre-trained attention layer;

[0238] The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

[0239] In one possible implementation, in the instructions executed by the processor 1010, pre-training the attention layer to be trained to obtain a pre-trained attention layer includes:

[0240] Determining a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and the video information has no contextual relationship;

[0241] Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and concatenating the fourth group of text features and the fourth group of noisy video features in chronological order to obtain a cascade feature;

[0242] The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features;

[0243] The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information. The loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

[0244] In one possible implementation, in the instructions executed by the processor 1010, the supervised training of the pre-trained attention layer to obtain the attention layer includes:

[0245] determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and the video clips of different scenes have a contextual relationship;

[0246] Encoding the text information to obtain a fifth set of text features, and encoding the video clip to obtain a fifth set of video features;

[0247] Noising the last set of video features in chronological order based on the original noise information with a preset first probability to obtain a set of noisy video features, concatenating the fifth set of text features, the fifth set of video features minus one set, and the set of noisy video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, using features in the weighted features corresponding to the last set of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information;

[0248] Noising all video features based on the original noise information with a preset second probability to obtain a fifth group of noisy video features, concatenating the fifth group of text features and the fifth group of video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information from the weighted features to obtain predicted noise information, and performing a second loss calculation based on the predicted noise information and the original noise information, where the sum of the first probability and the second probability is one;

[0249] Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

[0250] In a possible implementation, in the instructions executed by the processor 1010, the method further includes:

[0251] Get video information of multiple scenes;

[0252] For video information of different scenes, corresponding text description information is generated based on large language models;

[0253] The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

[0254] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the multi-scene video generation method described in any of the above embodiments.

[0255] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0256] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NAND FLASH), solid-state drives (SSDs)), etc.

[0257] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the multi-scene video generation method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0258] Based on the same inventive concept, corresponding to the multi-scene video generation method described in any of the above embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the multi-scene video generation method. For the execution entities corresponding to the steps in each embodiment of the multi-scene video generation method, the processors that execute the corresponding steps can belong to the corresponding execution entities.

[0259] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the multi-scene video generation method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0260] Those skilled in the art will appreciate that embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media containing computer-readable program code.

[0261] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive examples) of computer-readable storage media can include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0262] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0263] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0264] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0265] It should be understood that each block in the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine. These computer program instructions are executed by the computer or other programmable data processing device to produce a device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0266] These computer program instructions can also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner. In this way, the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0267] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0268] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in that particular order, or that all of the operations shown must be performed to achieve the desired results. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0269] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0270] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0271] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0272] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0273] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0274] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

[0275] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features of these aspects cannot be combined to benefit. Such division is merely for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A multi-scene video generation method, characterized in that: include: Determine prompt information input by the user, analyze the prompt information, and obtain an analysis result; In response to the analysis result that the prompt information is text information including a first number of scenes, encoding the text information to obtain a first number of groups of text features, and cascading the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain cascade features; Inputting the cascade features into the attention layer, performing attention mask processing on the cascade features based on the attention layer to obtain weighted features; using the feature corresponding to the original noise information in the weighted feature as the predicted noise information, and performing denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature; The denoising features are decoded to obtain video information including a first number of scenes.

2. The method according to claim 1, characterized in that The performing attention mask processing on the cascade features based on the attention layer includes: Dividing the concatenated features into the first number of groups of text features and the first number of groups of original noise information; The text features and the original noise information are respectively used as query parameters, corresponding key parameters and value parameters are determined in the attention mask, and cross attention calculation is performed.

3. The method according to claim 1, characterized in that The method further comprises: In response to the analysis result that the prompt information is text information containing a second number of scenes and video information containing a third number of scenes, the text information is encoded to obtain a second number of groups of text features, and the video information is encoded to obtain a third number of groups of video features. The second number of groups of text features, the third number of groups of video features, and a group of original noise information are cascaded in chronological order to obtain cascade features, wherein the second number is greater than or equal to three, and the third number is the second number minus one.

4. The method according to claim 1, wherein Before encoding the text information, the method further includes: Expand the text information based on a large language model to obtain expanded text information; Then, encoding the text information includes: The expanded text information is encoded.

5. The method according to claim 1, wherein The decoding of the denoising feature to obtain video information containing a first number of scenes includes: Decoding the denoising features to obtain multiple image frame information; Arrange the multiple image frame information in chronological order to obtain video information containing a first number of scenes.

6. The method according to claim 1, characterized in that The training method of the attention layer includes: Get the attention layer to be trained; Pre-training the attention layer to be trained to obtain a pre-trained attention layer; The pre-trained attention layer is subjected to supervised training to obtain the attention layer.

7. The method according to claim 6, characterized in that The pre-training of the attention layer to be trained to obtain a pre-trained attention layer includes: Determining a fourth number of video information containing a single scene and text information corresponding to each video information, wherein the fourth number is greater than or equal to three, and the video information has no contextual relationship; Encoding the text information to obtain a fourth group of text features, encoding the video information to obtain a fourth group of video features, performing noise processing on all video features based on the original noise information to obtain a fourth group of noisy video features, and concatenating the fourth group of text features and the fourth group of noisy video features in chronological order to obtain a cascade feature; The cascade features are input into the attention layer to be trained, and the cascade features are subjected to attention mask processing based on the attention layer to be trained to obtain weighted features; The features corresponding to the text information in the weighted features are removed to obtain the predicted noise information. The loss is calculated based on the predicted noise information and the original noise information. The parameters of the attention layer to be trained are updated based on the loss calculation to obtain the pre-trained attention layer.

8. The method according to claim 6, characterized in that The supervised training of the pre-trained attention layer to obtain the attention layer includes: determining video information including a fifth number of scenes and text information corresponding to a video clip of each scene, wherein the fifth number is greater than or equal to three, and the video clips of different scenes have a contextual relationship; Encoding the text information to obtain a fifth set of text features, and encoding the video clip to obtain a fifth set of video features; Noising the last set of video features in chronological order based on the original noise information with a preset first probability to obtain a set of noisy video features, concatenating the fifth set of text features, the fifth set of video features minus one set, and the set of noisy video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, using features in the weighted features corresponding to the last set of video features in chronological order as predicted noise information, and performing a first loss calculation based on the predicted noise information and the original noise information; Noising all video features based on the original noise information with a preset second probability to obtain a fifth group of noisy video features, concatenating the fifth group of text features and the fifth group of video features in chronological order to obtain cascaded features, inputting the cascaded features into a pre-trained attention layer, performing attention masking on the cascaded features based on the pre-trained attention layer to obtain weighted features, removing features corresponding to the text information from the weighted features to obtain predicted noise information, and performing a second loss calculation based on the predicted noise information and the original noise information, where the sum of the first probability and the second probability is one; Based on the first loss calculation and the second loss calculation, the parameters of the pre-trained attention layer are updated to obtain the attention layer.

9. The method according to claim 6, characterized in that The method further comprises: Get video information of multiple scenes; For video information of different scenes, corresponding text description information is generated based on large language models; The video information of the different scenes and the corresponding text description information are used as a test set to test the attention layer.

10. A multi-scene video generation device, characterized in that: include: a prompt information analysis module configured to determine prompt information input by a user, analyze the prompt information, and obtain an analysis result; an encoding and noise adding module, configured to, in response to the analysis result indicating that the prompt information is text information containing a first number of scenes, encode the text information to obtain a first number of groups of text features, and concatenate the first number of groups of text features with the first number of groups of original noise information in chronological order to obtain concatenated features; an attention mask module, configured to input the cascade features into an attention layer, and perform attention mask processing on the cascade features based on the attention layer to obtain weighted features; a denoising and restoration module configured to use a feature in the weighted feature corresponding to the original noise information as predicted noise information, and perform denoising processing on the original noise information based on the predicted noise information to obtain a denoised feature; The decoding module is configured to decode the denoising feature to obtain video information containing a first number of scenes.

11. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 9.

Images