A video generation method and related apparatus

Abstract

This application discloses a video generation method and related apparatus, comprising: generating a first video frame sequence corresponding to a first prompt word in response to a first video generation request; receiving a second video generation request, the second video generation request carrying a second prompt word and second time information, the second video frame sequence being the next video frame sequence after the first video frame sequence; generating a first attention parameter based on the first video frame sequence and the second prompt word carried in the second video generation request; determining a second attention parameter based on the first video frame sequence and the first prompt word; fusing the first attention parameter and the second attention parameter to obtain a target attention parameter; and generating a second video frame sequence based on the target attention parameter and the first attention parameter, which can improve the continuity of video frames.

Description

Technical Field

[0001] This application relates to the field of video processing, and more particularly to a video generation method and related apparatus. Background Technology

[0002] The video generation model generates corresponding video content based on the user's input text description and reference images. Users can dynamically adjust the generated video content, such as visual elements like style, main subject, scene environment, and action performance, by switching prompts. After switching prompts, if the semantic difference between the old and new prompts is too large, unreasonable abrupt changes may occur between consecutive video frames, resulting in poor video continuity.

[0003] In summary, a video generation method is needed that can improve the continuity of video footage. Summary of the Invention

[0004] This application provides a video generation method and related apparatus that can improve the continuity of video footage.

[0005] This application provides a video generation method, including: In response to a first video generation request, a first video frame sequence corresponding to a first prompt word is generated, wherein the first video generation request carries a first prompt word and first time information, and the first time information is used to indicate a first time interval corresponding to the first video frame sequence; A second video generation request is received, wherein the second video generation request carries a second prompt word and second time information, the second time information being used to indicate a second time interval corresponding to the second video frame sequence, the second time interval being the next time interval adjacent to the first time interval; Generate first attention parameters based on the first video frame sequence and the second cue word carried in the second video generation request; The first attention parameter and the second attention parameter are fused to obtain the target attention parameter, wherein the second attention parameter is determined based on the first cue word and the first video frame sequence; A second video frame sequence is generated based on the target attention parameters and the first attention parameters, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameters.

[0006] Another aspect of this application provides a video generation apparatus, comprising: The first request processing module is configured to respond to the first video generation request and generate a first video frame sequence corresponding to the first prompt word, wherein the first video generation request carries the first prompt word and first time information, and the first time information is used to indicate a first time interval corresponding to the first video frame sequence. The second request processing module is used to respond to receiving a second video generation request, wherein the second video generation request carries a second prompt word and second time information, the second time information is used to indicate a second time interval corresponding to the second video frame sequence, and the second time interval is the next time interval adjacent to the first time interval; The attention parameter generation module is used to generate first attention parameters based on the first video frame sequence and the second cue word carried in the second video generation request; The attention parameter fusion module is used to fuse the first attention parameter and the second attention parameter to obtain the target attention parameter, wherein the second attention parameter is determined based on the first cue word and the first video frame sequence; The video frame sequence generation module is used to generate a second video frame sequence based on the target attention parameters and the first attention parameters, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameters.

[0007] In one possible design, in another implementation of the embodiments of this application, the attention parameter fusion module is further used for: Determine the semantic association parameters between the first and second prompt words; Based on the semantic association parameters, determine the weight values ​​corresponding to the first attention parameter and the second attention parameter; Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0008] In one possible design, in another implementation of the embodiments of this application, the attention parameter fusion module is further used for: Based on the learnable matrix, the semantic association parameters are processed to obtain the weight values ​​corresponding to the first value vector; Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0009] In one possible design, in another implementation of the embodiments of this application, the attention parameter fusion module is further used for: If the frame number of the target frame is less than the preset frame number threshold, the transition coefficient is determined based on the frame number of the target frame, where the target frame is the last generated video frame in the second video frame sequence. Based on the transition coefficient, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined. The weight value corresponding to the first attention parameter is positively correlated with the transition coefficient, and the weight value corresponding to the first attention parameter is negatively correlated with the transition coefficient. Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0010] In one possible design, in another implementation of the embodiments of this application, the attention parameter fusion module is further used for: The transition coefficients are smoothed using a preset function to obtain the weight values ​​corresponding to the first value vector. The first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficients. Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0011] In one possible design, in another implementation of the embodiments of this application, the attention parameter generation module is further configured to: Generate visual vectors based on the first video frame sequence; Generate a second text vector based on the second prompt word; Linear projection processing is performed on the concatenated visual vector and the second text vector to obtain the first attention parameters.

[0012] In one possible design, in another implementation of the embodiments of this application, the attention parameter generation module is further configured to: The first time interval is divided into multiple sub-time intervals; In the first video frame sequence, at least one video frame is extracted from each sub-time interval based on the frame extraction frequency corresponding to each sub-time interval to obtain a reference video frame sequence. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0013] In one possible design, in another implementation of the embodiments of this application, the attention parameter generation module is further configured to: Based on the second cue word, identify the target object; For each video frame in the first video frame sequence, target detection is performed on the video frame to obtain the detection result corresponding to the video frame; Based on the detection results corresponding to each video frame, video frames whose detection results contain the target object are extracted from the first video frame sequence to obtain a reference video frame sequence; The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0014] In one possible design, in another implementation of the embodiments of this application, the attention parameter generation module is further configured to: For each video frame in the first video frame sequence, the area of ​​each target detection box in the video frame is determined based on the coordinates of each target detection box in the detection result corresponding to the video frame. For each video frame in the first video frame sequence, the relevance index corresponding to the video frame is determined based on the area and confidence of each target detection box in the video frame; Based on the relevance index corresponding to each video frame, the K video frames with the largest relevance index or video frames with relevance index exceeding the relevance threshold are extracted from the first video frame sequence to obtain the reference video frame sequence, where K is an integer greater than or equal to 1.

[0015] In one possible design, in another implementation of the embodiments of this application, the attention parameter generation module is further configured to: Based on the number of historical frames, the minimum total number of frames is determined. The number of historical frames is determined based on the total number of video frames in the first video frame sequence. The number of historical frames is positively correlated with the minimum total number of frames. Based on the minimum total number of frames, N video frames are extracted from the first video frame sequence to obtain a reference video frame sequence, where N is an integer greater than or equal to the minimum total number of frames. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0016] In one possible design, in another implementation of the embodiments of this application, a request generation module is further included, the request generation module being used for: Display the video generation interface; In response to a text input operation on the video generation interface, obtain the second prompt word; In response to a time input operation on the video generation interface, obtain second time information; Based on the second prompt word and the second time information, a second video generation request is generated.

[0017] In one possible design, in another implementation of the embodiments of this application, the video frame sequence generation module is further configured to: Based on the target attention parameters, generate the first K video frames in the second video frame sequence, where K is an integer greater than or equal to 1.

[0018] Based on the first attention parameter, generate the K+i-th video frame in the second video frame sequence, where i is an integer greater than or equal to 1.

[0019] Another aspect of this application provides a computer device, comprising: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute programs in memory, including methods for performing the aspects mentioned above; Bus systems are used to connect memory and processor to enable communication between them.

[0020] Another aspect of this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0021] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the above aspects.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: In response to a first video generation request, a first video frame sequence corresponding to a first prompt word is generated; a second video generation request is received, indicating that the video to be generated needs to switch from the first prompt word to the second prompt word; based on the first video frame sequence and the second prompt word carried in the second video generation request, a first attention parameter is generated; the first video frame sequence represents the generated portion of the video to be generated, and the first attention parameter encodes the understanding of the generated portion of the video to be generated in the context of the new prompt word; while the second attention parameter is determined based on the first prompt word and the first video frame sequence, encoding the understanding of the generated portion of the video to be generated in the context of the old prompt word; the first attention parameter and the second attention parameter are fused to obtain a target attention parameter, which incorporates the understanding of the generated portion of the video to be generated in different prompt word contexts; based on the target attention parameter and the first attention parameter, a second video frame sequence is generated; wherein at least one video frame in the second video frame sequence adjacent to the first video frame sequence is generated based on the target attention parameter, which can smoothly transition from the context of the old prompt word to the context of the new prompt word, improving the continuity of the video frame. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of an implementation environment for the video generation method in this application embodiment; Figure 2 This is a flowchart illustrating a video generation method in an embodiment of this application; Figure 3 This is a schematic diagram of the first time interval in an embodiment of this application; Figure 4 This is another schematic diagram of the first time interval in the embodiments of this application; Figure 5 This is a schematic diagram of a cross-attention calculation process in an embodiment of this application; Figure 6 This is a schematic diagram of a video generation window in an embodiment of this application; Figure 7 This is a schematic diagram of a reference video frame sequence in an embodiment of this application; Figure 8 This is a schematic diagram of the process for generating attention parameters in an embodiment of this application; Figure 9 This is another flowchart illustrating the generation of attention parameters in an embodiment of this application; Figure 10 This is a schematic diagram of a second video frame sequence in an embodiment of this application; Figure 11 This is a schematic diagram of an image processing apparatus in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of a computer device in an embodiment of this application. Detailed Implementation

[0024] This application provides a video generation method and related apparatus that can improve the continuity of video footage.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0028] It is understood that, in the specific embodiments of this application, user identification information, settlement data, and other related data require user permission or consent before being used in specific products or technologies. That is, before collecting user data, users can be notified through prompts, pop-ups, or voice prompts. The process of collecting user data only begins after obtaining user permission or consent. In other words, all user data collected in this application is collected with the user's consent, and the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0029] In essence, a video generation model refers to a type of model built upon deep learning and generative artificial intelligence technologies. It can automatically synthesize video sequences with continuous visual content and semantic consistency based on multimodal input information such as user-input text descriptions, reference images, audio signals, and keyframe sketches. This is achieved through temporal modeling, image generation, and inter-frame association learning, realizing end-to-end generation from abstract instructions to dynamic video content. Cue words are text instructions given by the user to the video generation model, specifying the video's main subject, scene environment, visual style, actions, lighting and color tones, camera language, composition, and other visual elements. This allows the visual generation model to understand the generation goal and constrain the generation direction. Users can dynamically adjust the generated video content, such as visual style, main subject, scene environment, and actions, by switching cue words. However, if the semantic difference between the old and new cue words is too large after switching, unreasonable abrupt changes may occur between consecutive video frames, resulting in poor video continuity.

[0030] Based on this, this application provides a video generation method that can improve the continuity of video frames. Specifically, in response to a first video generation request, a first video frame sequence corresponding to a first prompt word is generated; a second video generation request is received, indicating that the video to be generated needs to switch from the first prompt word to the second prompt word; based on the first video frame sequence and the second prompt word carried in the second video generation request, a first attention parameter is generated; the first video frame sequence represents the already generated part of the video to be generated, and the first attention parameter encodes the understanding of the already generated part of the video to be generated in the context of the new prompt word; while the second attention parameter is determined based on the first prompt word and the first video frame sequence, and encodes the understanding of the already generated part of the video to be generated in the context of the old prompt word; the first attention parameter and the second attention parameter are fused to obtain a target attention parameter, which incorporates the understanding of the already generated part of the video to be generated in different prompt word contexts; based on the target attention parameter and the first attention parameter, a second video frame sequence is generated; wherein at least one video frame in the second video frame sequence adjacent to the first video frame sequence is generated based on the target attention parameter, which can smoothly transition from the old prompt word context to the new prompt word context, improving the continuity of video frames.

[0031] Before introducing the specific methods of this application, the application scenarios of this application will be illustrated by example. It should be understood that the following application scenarios are merely illustrative and are not limited to these examples.

[0032] In some embodiments, the video generation method provided in this application is applicable to interactive real-time video generation scenarios. In this scenario, users need to modify prompts and add new creative intentions in real time during video generation to achieve dynamically controllable content generation. However, real-time prompt changes can cause drastic changes in the model's attention context, easily leading to abrupt scene transitions, subject disappearance, or morphological distortion. The solution in this application generates target attention parameters that fuse historical video frame sequences understood under different prompt contexts when the prompts are updated. This serves as a new attention context for video generation, enabling smooth transitions based on historical frame information and improving the continuity of the video footage.

[0033] In other embodiments, this method is also applicable to virtual roaming scenarios. In such scenarios, users need to dynamically switch scene descriptions, perspective content, or environmental styles during continuous roaming to achieve immersive and editable virtual space roaming. However, with real-time changes in prompts, problems such as abrupt scene splicing, disordered spatial structure, and abrupt perspective transitions can easily occur, affecting the smoothness of the roaming experience. The solution in this application generates target attention parameters that integrate historical virtual scene information understood in different scene description contexts when scene descriptions are switched. These parameters serve as new attention contexts for generating virtual scene information, enabling smooth transitions based on historical virtual scene information and improving the continuity of the virtual scene visuals.

[0034] It should be noted that the above application scenarios are merely examples, and the video generation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0035] The method provided in this application can be applied to... Figure 1 The illustrated implementation environment includes a terminal 110 and a server 120, and the terminal 110 and server 120 can communicate with each other via a communication network 130. The communication network 130 uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0036] The terminal 110 involved in this application includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, etc. The client is deployed on the terminal 110 and can run on the terminal 110 via a browser or as a standalone application (APP).

[0037] The server 120 involved in this application can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and AI platforms.

[0038] In the above implementation environment, in step S1, server 120 responds to the first video generation request sent by terminal 110 and generates a first video frame sequence corresponding to the first prompt word. The first video generation request carries the first prompt word and first time information, which indicates a first time interval corresponding to the first video frame sequence. In step S2, server 120 receives a second video generation request sent by terminal 110. The second video generation request carries a second prompt word and second time information, which indicates a second time interval corresponding to the second video frame sequence. The second time interval is the next time interval adjacent to the first time interval. In step S3, server 120... Server 120 generates a first attention parameter based on the first video frame sequence and the second prompt word carried in the second video generation request; in step S4, server 120 fuses the first attention parameter and the second attention parameter to obtain a target attention parameter, wherein the second attention parameter is determined based on the first prompt word and the first video frame sequence; in step S5, server 120 generates a second video frame sequence based on the target attention parameter and the first attention parameter, wherein at least one video frame in the second video frame sequence adjacent to the first video frame sequence is generated based on the target attention parameter; in step S6, server 120 returns the second video frame sequence to terminal 110.

[0039] Based on the above introduction, the video generation method in this application will be described below. Please refer to [link / reference]. Figure 2 In this application embodiment, the video generation method can be completed independently by server 120, or it can be completed in cooperation with terminal 110. The method of this application includes: 210. In response to the first video generation request, generate a first video frame sequence corresponding to the first prompt word, wherein the first video generation request carries the first prompt word and first time information, and the first time information is used to indicate the first time interval corresponding to the first video frame sequence; In one or more embodiments, a first video generation request refers to request information used to trigger a video generation model to generate a video segment from the video to be generated. Understandably, the video to be generated consists of multiple video segments arranged in chronological order. The first video generation request may request the generation of the first video segment in the video to be generated, or the second video segment, etc., without limitation. See also Figure 3 , Figure 3 This illustrates the scenario where the first video frame sequence is the first video segment in the video to be generated. See also... Figure 4 , Figure 4 This illustrates the scenario where the first video frame sequence is the second video segment in the video to be generated.

[0040] Specifically, the video generation model can be a video diffusion model. The video diffusion model architecture typically uses 3D convolution or a combination of 2D convolution and temporal attention mechanisms to capture the spatial and temporal dependencies in the video, thereby generating a sequence of video frames.

[0041] In one or more embodiments, the first video generation request may further include user-input images, audio, and data items such as video resolution parameters, frame rate parameters, and prompt word guidance intensity parameters.

[0042] In one or more embodiments, the first cue word refers to textual descriptive information used to guide the video generation model in generating the first video frame sequence. Specifically, the first cue word may specify the scene, style, color, composition, main subject, and main action of the video content.

[0043] In one or more embodiments, the first time information refers to the identification parameter that identifies the position of the first video frame sequence on the timeline of the video to be generated, and is used to locate the start and end times of the first video frame sequence, i.e., the first time interval.

[0044] In one or more embodiments, the first video frame sequence refers to the set of video frames generated by the video generation model in response to the first video generation request, based on a first prompt word. For example, the first prompt word is "city dusk, traffic flowing, high-rise glass reflecting the sunset glow," and the first time information is "start time 0s, end time 2s, frame rate 30fps." Based on this first prompt word and the first time information, the resulting first video frame sequence includes 60 consecutive frames within 0 to 2 seconds, sequentially presenting a city at dusk, traffic movement, and the changing light and shadow of the sunset.

[0045] 220. Receive a second video generation request, wherein the second video generation request carries a second prompt word and second time information, the second time information is used to indicate a second time interval corresponding to the second video frame sequence, and the second time interval is the next time interval adjacent to the first time interval; In one or more embodiments, the second video generation request refers to request information used to trigger a video generation model to generate the next video segment adjacent to the first video frame sequence. For example, see [link to relevant documentation]. Figure 3 and Figure 4 When the first video frame sequence is the first video segment in the video to be generated, the second video frame sequence is the second video segment in the video to be generated; when the first video frame sequence is the second video segment in the video to be generated, the second video frame sequence is the third video segment in the video to be generated.

[0046] In one or more embodiments, the second video generation request may further include user-input images, audio, and data items such as video resolution parameters, frame rate parameters, and prompt word guidance intensity parameters.

[0047] In one or more embodiments, the second cue word refers to textual descriptive information used to guide the video generation model in generating a second video frame sequence. Specifically, the second cue word may specify the scene, style, color, composition, main subject, and main action of the video content.

[0048] In one or more embodiments, the second time information refers to an identification parameter used to identify the position of the second video frame sequence on the timeline of the video to be generated, used to locate the start and end times of the second video frame sequence, i.e., the second time interval. The second time interval refers to the specific start and end range of the second video frame sequence on the timeline, and this interval is the next adjacent time interval of the first time interval. Specifically, when the first time interval is 0s-2s, the second time interval can be 2s-4s.

[0049] In one or more embodiments, the second video frame sequence refers to the set of video frames generated by the video generation model in response to a second video generation request, based on a second prompt word. For example, the second prompt word is "Night falls, streetlights turn on one by one, and traffic trails red taillights," and the second time information is "start time 2s, end time 4s, frame rate 30fps." The desired second video frame sequence is a set of 60 consecutive frames within 2 to 4 seconds, sequentially presenting the gradual darkening of the sky, the turning on of streetlights, and the changing light trails of traffic.

[0050] 230. Generate first attention parameters based on the first video frame sequence and the second cue word carried in the second video generation request; In one or more embodiments, the first attention parameter is a feature representation obtained by linearly transforming the first video frame sequence and the second cue word.

[0051] 240. The first attention parameter and the second attention parameter are fused to obtain the target attention parameter, wherein the second attention parameter is determined based on the first cue word and the first video frame sequence; In one or more embodiments, the second attention parameter is a feature representation obtained by linearly transforming the first video frame sequence and the first cue word.

[0052] In one or more embodiments, the target attention parameter refers to the feature representation obtained by fusing the first attention parameter and the second attention parameter, which is used to guide the video generation model to generate video frames within the second time interval, so that the generated video frames can simultaneously inherit the visual information of the first video frame sequence and reflect the semantic guidance of the second prompt word.

[0053] In one or more embodiments, the second attention parameter may be read from a cache or generated in real time by a computational task triggered by a second video generation request; no limitation is imposed here.

[0054] Understandably, video generation models utilize cross-attention mechanisms to inject conditional information. See also... Figure 5 The externally input conditional information (i.e., the conditional feature sequence) is projected as the source sequence for generating information into key vectors and value vectors. The video latent representation (i.e., the video spatiotemporal feature sequence) processed in the video generation model's decoding layer is used as the target sequence for receiving information, which is then queried by the projection layer (i.e., Figure 5 The Q-projection layer shown is projected onto the query vector. In this embodiment, two source sequences are set: the first source sequence contains a first video frame sequence and a second prompt word, and the second source sequence contains the first video frame sequence and the first prompt word. Two feature representations, namely a first attention parameter and a second attention parameter, are generated using these two source sequences respectively.

[0055] The first video frame sequence represents the generated video content, containing rich visual information such as scene, objects, and motion states, and is the foundation for maintaining the temporal continuity of the video. The first attention parameter encodes the understanding of the first video frame sequence in the context of the second cue word, that is, the understanding of the generated video content in the context of the current cue word. The second attention parameter encodes the understanding of the first video frame sequence in the context of the first cue word, that is, the understanding of the generated video content in the context of the previous cue word. The target attention parameter, obtained by fusing the first and second attention parameters, can integrate the understanding of the generated video content in different cue word contexts. By injecting the old semantics from the first cue word and the new semantics from the second cue word into the target sequence through the target attention parameter, the new semantics do not overwhelmingly disrupt the established visual continuity. It inherits the visual continuity information of the historical images while injecting the semantic guidance of the new cue word, thereby guiding the generation of subsequent frames.

[0056] Specifically, the first attention parameters include a first key vector and a first value vector. The first key vector is the feature representation obtained by linearly transforming each element in the first source sequence. As an index, the first key vector contains a summary of the content of each element in the source sequence and is used to match it with the query vector of the target sequence. By calculating the similarity between the query vector and the key vector, the video generation model can determine which positions in the source sequence are most correlated with the current position of the target sequence. The first value vector is the feature representation obtained by performing another independent linear transformation on each element in the first source sequence. The first value vector carries the actual semantic information of each element in the first source sequence.

[0057] Specifically, the second attention parameters include a second key vector and a second value vector. The second key vector is the feature representation obtained by linearly transforming each element in the second source sequence. As an index, the second key vector contains a summary of the content of each element in the source sequence and is used to match it with the query vector of the target sequence. By calculating the similarity between the query vector and the key vector, the video generation model can determine which positions in the source sequence are most correlated with the current position of the target sequence. The second value vector is the feature representation obtained by performing another independent linear transformation on each element in the second source sequence. The second value vector carries the actual semantic information of each element in the second source sequence. Attention weights are calculated by matching the target key vector obtained by fusing the query vector, the first key vector, and the second key vector. These attention weights are used to perform a weighted summation on the target value vector obtained by fusing the first value vector and the second value vector, and video frames are generated based on the weighted summation result.

[0058] 250. Generate a second video frame sequence based on the target attention parameters and the first attention parameters, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameters.

[0059] In one or more embodiments, at least one transitional video frame adjacent to the first video frame sequence in the second video frame sequence is generated based on the target attention parameter. The transitional video frame integrates the understanding of the historical scene content in the context of the first cue word and the second cue word to achieve a smooth change in the semantics of the scene.

[0060] In one or more embodiments, the remaining video frames in the second video frame sequence, excluding transitional video frames, are generated based on a first attention parameter, and these video frames follow the semantic guidance of a second cue word.

[0061] This application provides a method for video generation. In response to a first video generation request, a first video frame sequence corresponding to a first prompt word is generated; a second video generation request is received, indicating that the video to be generated needs to switch from the first prompt word to the second prompt word; based on the first video frame sequence and the second prompt word carried in the second video generation request, a first attention parameter is generated; the first video frame sequence represents the already generated portion of the video to be generated, and the first attention parameter encodes the understanding of the already generated portion of the video to be generated in the context of the new prompt word; while the second attention parameter is determined based on the first prompt word and the first video frame sequence, encoding the understanding of the already generated portion of the video to be generated in the context of the old prompt word; the first attention parameter and the second attention parameter are fused to obtain a target attention parameter, which incorporates the understanding of the already generated portion of the video to be generated in different prompt word contexts; based on the target attention parameter and the first attention parameter, a second video frame sequence is generated; wherein at least one video frame in the second video frame sequence adjacent to the first video frame sequence is generated based on the target attention parameter, which can smoothly transition from the old prompt word context to the new prompt word context, improving the continuity of the video frame.

[0062] Optionally, in the above Figure 2 Based on the corresponding embodiments, in the first optional embodiment of the video generation method provided in this application, the video generation method further includes: Display the video generation interface; In response to a text input operation on the video generation interface, obtain the second prompt word; In response to a time input operation on the video generation interface, obtain second time information; Based on the second prompt word and the second time information, a second video generation request is generated.

[0063] As can be seen from the foregoing embodiments, the second prompt word refers to the textual description information used to guide the video generation model in generating the second video frame sequence. The second time information refers to the identification parameters used to identify the position of the second video frame sequence on the timeline of the video to be generated.

[0064] See Figure 6 The video generation interface refers to a graphical user interface provided to users for interactive video generation. It receives prompts, timing information, and other generation parameters input by the user. Users can enter prompts through text input boxes, specify the timing of prompt switching through time sliders or numerical input boxes, and preview the generated video content in real time.

[0065] In one or more embodiments, a second video generation request can be generated by collecting user input through a front-end video generation interface, constructing a structured request (such as JSON format), and sending it to a back-end video generation service; alternatively, a second video generation request can be generated by calling the API function of a local video generation model and directly passing in a second prompt word and second time information; no limitation is imposed here.

[0066] This application provides a method for video generation. By displaying a video generation interface and responding to text input and time input operations on the interface, a second prompt word and second time information are obtained to generate a second video generation request. This method supports dynamic prompt word switching in interactive video generation scenarios, allowing users to intervene in the content's direction in real time during video generation, achieving an interactive experience of creation while generating. Simultaneously, it provides necessary semantic and temporal conditions for subsequent generation of transition frames based on the second prompt word.

[0067] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, a first attention parameter is generated according to the first video frame sequence and the second cue word carried in the second video generation request, including: The first time interval is divided into multiple sub-time intervals; In the first video frame sequence, at least one video frame is extracted from each sub-time interval based on the frame extraction frequency corresponding to each sub-time interval to obtain a reference video frame sequence. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0068] In one or more embodiments, a sub-time interval refers to multiple non-overlapping time periods obtained by dividing a first time interval, and the multiple sub-time intervals together constitute a part or all of the first time interval.

[0069] In one or more embodiments, the interval lengths of the various sub-time intervals may be equal or unequal, and no limitation is imposed here.

[0070] In one or more embodiments, the frame extraction frequency refers to the number of video frames extracted per unit time. A higher frame extraction frequency indicates that the video frames extracted within that sub-time interval are more dense, and the visual change information within that interval is more fully preserved; a lower frame extraction frequency indicates that the video frames extracted are sparser, and only the most representative keyframes within that interval are retained.

[0071] In one or more embodiments, the reference video frame sequence refers to a set of frames selected from various sub-time intervals, and the reference video frame sequence characterizes the trend of visual content change within the first time interval.

[0072] In one or more embodiments, the number of video frames extracted from each sub-time interval may be equal or unequal.

[0073] In one or more embodiments, at least one video frame may be extracted from each sub-time interval based on a variability index, or at least one video frame may be extracted from each sub-time interval based on the detection results of object detection, without limitation.

[0074] Specifically, a frame vector is generated for each video frame within the sub-time interval. Based on the frame vectors, the vector distance between the video frame and its neighboring frames is calculated to obtain the variability index corresponding to the video frame. The K video frames with the largest variability indices are extracted as the extraction results for that sub-time interval.

[0075] Understandably, the reference video frame contains video frames from each sub-time interval, which can retain the main visual change information of the first video frame sequence and reduce the number of redundant video frames with highly similar visual content. This helps to reduce computational complexity in subsequent attention generation and reduce the interference of redundant video frames on attention generation.

[0076] In one or more embodiments, different frame-sampling frequencies are used for each sub-time interval, and the sampling frequency increases chronologically. See also Figure 7 The further away from the second time interval the frame sampling frequency is, the lower the frame sampling frequency of the sub-time interval, and the closer to the second time interval the frame sampling frequency is, the more the reference video frame sequence focuses on representing the visual content of the later part of the first time interval, thus providing more recent visual information for the transition from the first cue word context to the second cue word context, improving the smoothness and semantic coherence of the transition frames.

[0077] For example, the above-mentioned multiple sub-time intervals include a first sub-time interval and a second sub-time interval whose timing follows the first sub-time interval; the length of the first sub-time interval is equal to the length of the second sub-time interval, and the frame sampling frequency of the first sub-time interval is lower than the frame sampling frequency of the second sub-time interval.

[0078] For example, the aforementioned multiple sub-time intervals include a first sub-time interval and a second sub-time interval whose timing follows the first sub-time interval; the length of the first sub-time interval is greater than the length of the second sub-time interval. K keyframes are extracted from the first sub-time interval and the second sub-time interval respectively to obtain a reference video frame sequence. The reference video frame sequence contains 2K reference video frames, and the proportion of video frames from the later part of the first time interval is higher in the reference video frame sequence.

[0079] This application provides a method for video generation. By dividing a first time interval into multiple sub-time intervals and extracting video frames from each sub-time interval based on the frame extraction frequency corresponding to that sub-time interval, a reference video frame sequence is obtained. This reference video frame sequence can characterize the main visual changes in the first video frame sequence. Based on the reference video frame sequence and a second cue word carried in a second video generation request, a first attention parameter is generated. This reduces the complexity of attention parameter calculation, minimizes interference from redundant video frames on attention generation, and optimizes the continuity of the second video frame sequence.

[0080] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, a first attention parameter is generated according to the first video frame sequence and the second cue word carried in the second video generation request, including: Based on the second cue word, identify the target object; For each video frame in the first video frame sequence, target detection is performed on the video frame to obtain the detection result corresponding to the video frame; Based on the detection results corresponding to each video frame, video frames whose detection results contain the target object are extracted from the first video frame sequence to obtain a reference video frame sequence; The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0081] In one or more embodiments, the target object refers to the visual subject parsed from the second prompt word, used to filter video frames related to the second prompt word from the first video frame sequence. The target object can be a person, vehicle, building, scene, etc., and is not limited here.

[0082] In one or more embodiments, the target object can be determined by keyword extraction or by named entity recognition; no limitation is made here.

[0083] In one or more embodiments, target detection refers to identifying and locating target objects in a video frame sequence, used to determine whether a video frame contains a target object, and the detection result includes the coordinates of the target detection box used to locate the target object, the category label, and the confidence score.

[0084] In one or more embodiments, video frames can be detected using deep learning-based object detection networks (such as YOLO or Faster R-CNN), or using end-to-end detection models based on Transformers (such as DETR). No limitation is imposed here.

[0085] Understandably, the reference video frame sequence is obtained by extracting video frames containing the target object from the first video frame sequence. It can retain visual information related to the target object, and the number of frames in the reference video frame sequence is less than the number of frames in the first video frame sequence. Therefore, it can reduce the computational complexity in subsequent attention generation and optimize the visual continuity of the target object in the second video frame sequence.

[0086] In this embodiment, the target object is first determined based on the second prompt word; then, target detection is performed on the video frames, and video frames containing the target object are extracted from the first video frame sequence to obtain a reference video frame sequence. The reference video frame sequence contains visual information related to the target subject; based on the reference video frame sequence and the second prompt word carried in the second video generation request, a first attention parameter is generated, which can reduce the complexity of attention parameter calculation and optimize the visual continuity of the target object in the second video frame sequence.

[0087] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, based on the detection results corresponding to each video frame, video frames whose detection results contain the target object are extracted from the first video frame sequence to obtain a reference video frame sequence, including: For each video frame in the first video frame sequence, the area of ​​each target detection box in the video frame is determined based on the coordinates of each target detection box in the detection result corresponding to the video frame. For each video frame in the first video frame sequence, the relevance index corresponding to the video frame is determined based on the area and confidence of each target detection box in the video frame; Based on the relevance index corresponding to each video frame, the K video frames with the largest relevance index or video frames with relevance index exceeding the relevance threshold are extracted from the first video frame sequence to obtain the reference video frame sequence, where K is an integer greater than or equal to 1.

[0088] As described in the foregoing embodiments, the target object refers to the visual subject parsed from the second prompt word; the detection result includes a target detection box used to locate the target object. Specifically, the detection result includes the coordinates of the target detection box and the confidence score of the target detection box.

[0089] In one or more embodiments, a target detection box refers to a bounding rectangle in a video frame that marks the location of a target object.

[0090] In one or more embodiments, the relevance index is an indicator used to quantify the degree of association between a single video frame and a target object, and is used to determine whether the video frame should be selected into the reference video frame sequence.

[0091] Understandably, the area of ​​the detection box reflects the proportion of the target object in the frame, which helps ensure that the selected reference frame contains a clear and complete target subject; the confidence score of the target detection box reflects the completeness and clarity of the target object in the frame. Based on the area and confidence score of the target detection box, the relevance index can filter out video frames where the target object is too small (low area proportion) or blurry (low confidence score). It can eliminate invalid frames and retain effective visual information that is highly relevant to the second cue word.

[0092] In one or more embodiments, the relevance index corresponding to a video frame can be determined by a weighted calculation method based on the area of ​​the target detection box and the confidence level; the relevance index corresponding to a video frame can also be determined by the area ratio of the screen occupied by the target detection box and the number of target detection boxes, without any limitation.

[0093] For example, if the target object parsed from the second prompt word is a pedestrian, after frame-by-frame detection of the first video frame sequence: if a pedestrian is detected in a certain video frame, the target detection box area is large and the confidence score is 0.92, then the relevance index of this frame is high and it is preferentially extracted as a reference video frame; if a pedestrian is detected in another video frame, but the detection box area is too small, the target is blurry, and the confidence score is 0.35, then the relevance index of this frame is low and it is not selected into the reference video frame sequence.

[0094] This application provides a method for video generation. First, the area of ​​each target detection box is determined based on its coordinates. Second, a relevance index corresponding to each video frame is determined based on the area and confidence level of each target detection box in the video frame. Finally, the K video frames with the highest relevance index or video frames with relevance index exceeding the relevance threshold are extracted from the first video frame sequence. This process can filter out video frames where the target object is too small (low area ratio) or the target object is blurry (low confidence level), resulting in a more reliable reference video frame sequence.

[0095] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, a first attention parameter is generated according to the first video frame sequence and the second cue word carried in the second video generation request, including: The minimum total number of frames is determined based on the number of historical frames. The number of historical frames is determined based on the total number of video frames in the first video frame sequence. The number of historical frames is positively correlated with the minimum total number of frames. Based on the minimum total number of frames, N video frames are extracted from the first video frame sequence to obtain a reference video frame sequence, where N is an integer greater than or equal to the minimum total number of frames. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0096] In one or more embodiments, the number of historical frames indicates the number of video frames already generated in the video to be generated. Specifically, see [link to relevant documentation]. Figure 3 When the first video frame sequence is the first video segment in the video to be generated, the number of historical frames is the number of frames in the first video frame sequence. See also... Figure 4 When the first video frame sequence is the second video segment in the video to be generated, the number of historical frames is the sum of the number of frames in the first video frame sequence and the number of frames in the first video segment in the video to be generated. Figure 4 The total number of frames in the third video frame sequence, and so on.

[0097] In one or more embodiments, the minimum total number of frames refers to the minimum number of frames required to be satisfied by the reference video frame sequence, which is used to ensure that the attention calculation has sufficient historical information.

[0098] For example, the video generation model has generated 10 video frames and is generating the 11th video frame. Frames 1 to 9 are in a first time interval, and frame 10 is in a second time interval. The number of historical frames is 10, and the minimum total number of frames is determined to be 5.

[0099] For example, the video generation model has generated 48 video frames and is generating the 49th video frame. Frames 20 to 40 are in the first time interval, and frames 41 to 48 are in the second time interval. The number of historical frames is 48, and the minimum total number of frames is determined to be 10.

[0100] Understandably, a larger number of historical frames indicates more historical visual information available for calculating the current attention parameters. Therefore, by dynamically adjusting the minimum total number of frames based on the number of historical frames, we can reduce the introduction of irrelevant information and speed up the generation process in the early stages of video generation; and in the later stages of video generation, we can provide sufficient visual information for the current attention parameters to ensure video continuity.

[0101] In one or more embodiments, the minimum total number of frames can be determined by an exponential function or by a linear function; no limitation is made here.

[0102] In one or more embodiments, N video frames can be extracted from the first video frame sequence based on a relevance index, or N video frames can be extracted from the first video frame sequence based on a variability index, without limitation.

[0103] In one or more embodiments, a maximum total number of frames can also be set, where N is greater than or equal to the minimum total number of frames and N is less than or equal to the maximum total number of frames.

[0104] This application provides a method for video generation. First, based on the number of historical frames, a minimum total number of frames is determined. Then, video frames no less than the minimum total number of frames are extracted from a first video frame sequence to obtain a reference video frame sequence. The number of frames in the reference video frame sequence can be dynamically adjusted to provide sufficient visual information for attention parameter calculation and ensure video continuity.

[0105] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, a first attention parameter is generated according to the first video frame sequence and the second cue word carried in the second video generation request, including: Generate visual vectors based on the first video frame sequence; Generate a second text vector based on the second prompt word; Linear projection processing is performed on the concatenated visual vector and the second text vector to obtain the first attention parameters.

[0106] In one or more embodiments, a visual vector refers to a feature vector obtained by visual feature encoding of a first video frame sequence, used to characterize visual information such as the subject, scene, texture, and color in the video frame.

[0107] In one or more embodiments, the first text vector refers to a feature vector obtained by text semantic encoding based on the first prompt word, which is used to characterize the text semantics corresponding to the first prompt word.

[0108] In one or more embodiments, visual vectors can be generated using a pre-trained video feature coding network (such as ViT), without limitation.

[0109] In one or more embodiments, a second text vector can be generated using a pre-trained text encoding network (BERT), without limitation.

[0110] Similarly, see Figure 8 Based on the first cue word, a first text vector is generated; the concatenated visual vector and the first text vector are subjected to linear projection processing to obtain a second attention parameter containing a second value vector and a second key vector. Based on the second cue word, a second text vector is generated; the concatenated visual vector and the second text vector are subjected to linear projection processing to obtain a first attention parameter containing a first value vector and a first key vector.

[0111] Understandably, performing linear projection on the concatenated visual vector and the second text vector yields the first attention parameter, which aligns the visual content with the semantics of the second cue word into the same attention space. The generated first attention parameter (i.e., the key vector and the value vector) encodes the visual information of the first video frame sequence and the semantic guidance of the second cue word, which is beneficial for generating video frames that both follow the second cue word and continue the visual content of the first video frame sequence.

[0112] This application provides a method for video generation. First, based on a first video frame sequence and a second cue word, a visual vector and a second text vector are generated respectively. The two are then concatenated and linearly projected to obtain a first attention parameter. The first attention parameter encodes the visual information of the first video frame sequence and the semantic guidance of the second cue word, which is beneficial for generating video frames that both follow the second cue word and continue the visual content of the first video frame sequence.

[0113] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter, including: Determine the semantic association parameters between the first and second prompt words; Based on the semantic association parameters, determine the weight values ​​corresponding to the first attention parameter and the second attention parameter; Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0114] In one or more embodiments, the semantic association parameter refers to a parameter used to quantify the semantic relevance between the first and second prompt words. It measures the magnitude of semantic differences between the preceding and following prompt words in dimensions such as subject, scene, action, and style, providing a basis for adjusting the weighted fusion of attention parameters. This parameter can be expressed in the form of semantic similarity or semantic distance, without limitation here.

[0115] Specifically, when the semantic association parameter is in the form of semantic distance, the weight value corresponding to the first attention parameter is positively correlated with the semantic association parameter, and the weight value corresponding to the second attention parameter is negatively correlated with the semantic association parameter. When the semantic association parameter is in the form of semantic similarity, the weight value corresponding to the first attention parameter is negatively correlated with the semantic association parameter, and the weight value corresponding to the second attention parameter is positively correlated with the semantic association parameter.

[0116] Understandably, the greater the semantic distance between the first and second cue words, the greater the semantic difference between the two cue words, requiring more incorporation of the second cue word context (i.e., reducing reliance on original visual features). Conversely, the smaller the semantic distance between the first and second cue words, the closer their meanings, allowing for better preservation of visual features within the first cue word context. Therefore, by setting the weight values ​​corresponding to the first and second attention parameters, the contribution ratios of the original and new contexts can be balanced: when the semantic difference is large, increasing the weight of the first attention parameter and decreasing the weight of the second attention parameter strengthens the injection of new semantics; when the semantic difference is small, decreasing the weight of the first attention parameter and increasing the weight of the second attention parameter preserves more original visual features, ensuring a smooth and natural transition.

[0117] For example, the semantic changes vary greatly depending on the scenario of different prompts. For instance, "man walking → man running" represents a minor semantic change, while "man walking → a dragon appears in the sky" represents a major semantic change. In the former scenario, the video frame to be generated has a higher requirement for inheriting previous historical visual features. Therefore, a larger weight is allocated to the second attention parameter to more fully inherit visual features such as the character's appearance, posture, and environment from previous frames. Minor updates to action details are made based on the second prompt to ensure a smooth and natural flow between the character and the visuals. In the latter scenario, the requirement for inheriting historical visual features is significantly reduced. Therefore, a larger weight is allocated to the first attention parameter. While maintaining necessary sequential continuity, it more prominently generates entirely new visual content such as the dragon according to the new prompt, adaptively balancing the contradiction between inheriting historical features and generating new content.

[0118] In this embodiment of the application, a video generation method is provided. First, a semantic association parameter between a first prompt word and a second prompt word is determined. The semantic association parameter represents the semantic difference between the first prompt word and the second prompt word. Based on the semantic association parameter, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined, which can balance the importance of the first prompt word and the second prompt word in the generation of subsequent video frames, taking into account both video continuity and fidelity to the first prompt word.

[0119] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, determining the weight value corresponding to the first attention parameter and the weight value corresponding to the second attention parameter based on the semantic association parameter includes: Based on the learnable matrix, the semantic association parameters are processed to obtain the weight values ​​corresponding to the first value vector; Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0120] As can be seen from the foregoing embodiments, the semantic association parameter refers to the parameter that quantifies the semantic relevance between the first prompt word and the second prompt word.

[0121] In one or more embodiments, a learnable matrix refers to a nonlinear projection matrix (or neural network layer) containing trainable parameters, used to map semantic association parameters to weight values ​​corresponding to a first value vector.

[0122] In one or more embodiments, the learnable matrix may be a non-linear activation layer or a multilayer perceptron (MLP), without limitation herein.

[0123] Understandably, learnable matrices can adaptively learn the optimal mapping relationship between semantic association parameters and weight values, freeing weight allocation from manually designed rules or fixed functions, thus better adapting to the diversity of different video content and semantic variations of prompt words. For example, the semantic similarity between the first and second prompt words is 0.15, the weight value corresponding to the first key vector is 0.73, the weight value corresponding to the first value vector is 0.73, the weight value corresponding to the second key vector is 0.27, and the weight value corresponding to the second value vector is 0.27.

[0124] For example, the semantic similarity between the first prompt word and the second prompt word is 0.85, the weight value corresponding to the first key vector is 0.26, the weight value corresponding to the first value vector is 0.26, the weight value corresponding to the second key vector is 0.74, and the weight value corresponding to the second value vector is 0.74.

[0125] For example, see Figure 9 The video generation model includes a visual encoder, a text encoder, and a feature fusion mechanism based on cross-attention. The visual encoder converts the first video frame sequence into visual vectors, and the text encoder converts the first and second cue words into first and second text vectors, respectively. The feature fusion mechanism converts the first and second text vectors and the visual vectors into target value vectors and target key vectors, including a value projection layer (i.e., ...). Figure 9 The V-projection layer shown), the bond projection layer (i.e. Figure 9 The diagram shows a K-projection layer and a nonlinear activation layer. This nonlinear activation layer consists of a fully connected layer and a Sigmoid activation function.

[0126] This application provides a method for video generation. Based on a learnable matrix, semantic association parameters are processed to obtain weight values ​​corresponding to a first value vector. The learnable matrix can adaptively learn the optimal mapping relationship between semantic association parameters and weight values ​​to improve the quality of target attention parameters.

[0127] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter, including: If the frame number of the target frame is less than the preset frame number threshold, then the transition coefficient is determined based on the frame number of the target frame, wherein the target frame is the last generated video frame in the second video frame sequence. Based on the transition coefficient, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined. The weight value corresponding to the first attention parameter is positively correlated with the transition coefficient, and the weight value corresponding to the second attention parameter is negatively correlated with the transition coefficient. Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0128] Understandably, even with significant semantic changes, a single-frame abrupt change can still produce a subtle sense of unnaturalness. Therefore, see [link / reference]. Figure 10 In the second video frame sequence, multiple transition frames are set as transition stages, and the remaining video frames are ordinary frames, which are generated based on the first attention parameters.

[0129] In one or more embodiments, the target frame is the last generated video frame in the second video frame sequence. The preset frame number threshold refers to the total number of transition frames set in advance, that is, the number of frames required to completely switch from the first prompt word context to the second prompt word context: when the frame number of the target frame is less than the frame number threshold, progressive fusion is adopted, and the next video frame is generated using the target attention parameter; when it is equal to or greater than the threshold, the next video frame is generated using the first attention parameter.

[0130] In one or more embodiments, the transition coefficient is a scalar value between 0 and 1, which characterizes the degree of gradualness of the current generated frame in the transition phase. It is used to control the weight allocation of the first attention parameter and the second attention parameter when the current generated frame is generated, so as to achieve a smooth and gradual transition from the first context to the second context.

[0131] For example, the video generation model has generated two video frames in the second time interval. The frame number of the target frame is 2, the preset frame number threshold is 10, and the calculated transition coefficient is 0.2.

[0132] Understandably, if the frame number of the target frame is less than a preset frame number threshold, it indicates that the currently generated frame is still in the transition phase and has not yet completed the full switch from the context of the first prompt word to the context of the second prompt word. By controlling the degree of fusion of each transition frame through the transition coefficient, a gradual semantic change can be achieved in the time dimension, reducing the abruptness caused by hard switching of a single frame. Specifically, the larger the transition coefficient, the closer the currently generated frame is to the end of the transition phase, the higher the contribution weight of the second prompt word, and the lower the contribution weight of the first prompt word.

[0133] This application provides a method for video generation. When the frame number of the target frame is less than a preset frame number threshold, a transition coefficient is determined based on the frame number of the target frame. The transition coefficient characterizes the degree of gradual transition of the currently generated frame during the transition phase. Based on the transition coefficient, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined, which enables multi-frame transitions in the time dimension, reduces the visual unnaturalness caused by hard switching of a single frame, and balances video continuity with the accuracy of semantic switching.

[0134] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, the first attention parameter includes a first value vector and a first key vector; the second attention parameter includes a second value vector and a second key vector; determining the weight value corresponding to the first attention parameter and the weight value corresponding to the second attention parameter based on the transition coefficient includes: The transition coefficients are smoothed using a preset function to obtain the weight values ​​corresponding to the first value vector. The first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficients. Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0135] In one or more embodiments, smoothing refers to applying a non-linear transformation to the transition coefficients to make the weight value change curves smoother, achieving a gradual change in visual content during the transition process. This can be achieved using a monotonically increasing function with a negative second derivative.

[0136] In one or more embodiments, smoothing can be performed using nonlinear monotonically increasing functions such as the Sigmoid function, arctangent function, quadratic decreasing function, and logarithmic smoothing function; no limitation is imposed here.

[0137] For example, the first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficient. Assuming the second derivative of the preset function is greater than zero in the interval [0, 1], the function exhibits a downward convex shape, and its growth rate accelerates as the independent variable increases, achieving a gradual entry effect. Assuming the second derivative of the preset function is less than zero in the interval [0, 1], the function exhibits an upward convex shape, and its growth rate slows down as the independent variable increases, achieving a gradual exit effect.

[0138] For example, the first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficient. Assume that the second derivative of the preset function is greater than zero in the interval [0, M), equal to zero at the transition coefficient M, and less than zero in the interval (M, 1]. Then the function exhibits a downward convex shape in the interval [0, M] and an upward convex shape in the interval [M, 1], achieving a smooth in-and-out transition effect. The preset function is specifically as follows:

[0139] In the formula, Transition coefficient; Pi; This represents the weight value corresponding to the first value vector.

[0140] For example, the transition coefficient is 0.85, and after the smoothing function is processed, the weight value corresponding to the first value vector is 0.94, the weight value corresponding to the first key vector is 0.94, the weight value corresponding to the second value vector is 0.06, and the weight value corresponding to the second key vector is 0.06.

[0141] This application provides a method for video generation. By using a preset function to smooth the transition coefficients, the weight values ​​corresponding to the first value vector are obtained. This, in turn, determines the weight values ​​corresponding to the first key vector, the second key vector, and the second value vector, which helps to achieve a gradual in or gradual out transition effect.

[0142] Optionally, in the above Figure 2 Based on the corresponding embodiments, in another optional embodiment of the video generation method provided in this application, a second video frame sequence is generated according to the target attention parameters and the first attention parameters, including: Based on the target attention parameters, generate the first K video frames in the second video frame sequence, where K is an integer greater than or equal to 1; Based on the first attention parameter, generate the K+i-th video frame in the second video frame sequence, where i is an integer greater than or equal to 1.

[0143] As described in the foregoing embodiments, the first attention parameter refers to the feature representation obtained by linearly transforming the first video frame sequence and the first cue word, encoding the understanding of the first video frame sequence in the context of the second cue word. The target attention parameter refers to the feature representation obtained by fusing the first attention parameter and the second attention parameter, used to generate video frames within the second time interval, so that the generated video frames can simultaneously inherit the visual information of the first video frame sequence and reflect the semantic guidance of the second cue word.

[0144] For example, the video generation model uses an autoregressive approach to generate video. First, it calculates the first attention parameter, the second attention parameter, and the target attention parameter; then, it writes the calculated first attention parameter into a cache. Based on the target attention parameter, it generates the first K video frames in the second video frame sequence one by one; it reads the first attention parameter from the cache and generates the (K+i)th video frame in the second video frame sequence based on the first attention parameter, until it reaches the next prompt word switching node or the end node of the video to be generated.

[0145] Specifically, the video generation model uses the DiT model (Diffusion Transformer). This is based on the initial prompt words given to the user. The DiT model generates video frames using an autoregressive approach. When the control module detects that the user has changed the prompt word... Change to And the current generation pointer has reached the th Frame. Pause the regular generation process and activate the KV update module (KV-refresh module).

[0146] The KV-refresh module will generate the frame sequence. Input to visual encoder (For example, a pre-trained VAE encoder or a specialized video feature extractor) to obtain a set of visual vectors. .

[0147]

[0148] The KV-refresh module will add new prompt words. Input to text encoder (For example, the text encoding part of CLIP), resulting in a set of text vectors. .

[0149]

[0150] The KV-refresh module reads prompt words from the cache. Text feature vectors .

[0151] KV-refresh module: converts visual vectors Separately with text vectors , Concatenate the vectors to obtain the concatenated vector. , :

[0152]

[0153] The KV-refresh module will concatenate vectors. As conditional information, it is input into the cross-attention layer of the DiT model to obtain the first attention parameters, which include the first key vector. and the first value vector .

[0154] The KV-refresh module will concatenate vectors. As conditional information, it is input into the cross-attention layer of the DiT model to obtain the second attention parameters, which contain the second key vector. Second value vector .

[0155] Specifically, the cross-attention layer in DiT, originally used to receive text conditions, now receives... or As its context input, it is projected through the layer's built-in key-linear projection matrix. Sum linear projection matrix Calculate the first attention parameter and the second attention parameter:

[0156]

[0157]

[0158]

[0159] The KV-refresh module provides weight-based... The first attention parameter and the second attention parameter are weighted and fused to obtain a vector containing the target key. and target value vector Target attention parameters:

[0160]

[0161] The DiT model generates the first [number] consecutively based on the target attention parameters. Frame, First Frame sequence of frames (For ): That is, from the first At the start of the frame, the DiT model will use [method / process] during denoising and generation. and Transition frames are generated as context during the transition phase. Then, the DiT model uses this transition frame during denoising and generation. and The remaining normal video frames are generated as context.

[0162] In one or more embodiments, this application also provides a method for training a video generation model, comprising: Obtain the video sample F_full and its corresponding text description P_full. Randomly generate an integer segmentation point t_rand within the video sample F_full. Generate a new text description P_B for the video segment F_(t_rand: t_end) after t_rand, which can be generated with the assistance of a large language model or using existing multi-segment labeled data. The text description P_A for the video segment F_(t_start: t_rand) before t_rand can be all or part of P_full.

[0163] During the training loop, when the model processes the t_rand-th frame, it is forced to generate new attention parameters using F_(1:t_rand-1) and P_A, P_B. The loss function of the video generation model (such as the mean squared error in the latent space) penalizes generation results that fail to follow the description of P_B ​​after t_rand. Specifically, the difference between the video frames generated by the video generation model and the real video frames in the latent space is compared, and this difference is used as the loss value to participate in the back-updating of the model parameters. This forces the video generation model to make full use of the new attention parameters for video generation, improving the robustness of the video generation model to prompt word switching.

[0164] In one or more embodiments, this application also provides a system for implementing interactive video generation, comprising: The user interface provides users with a visual creative environment, specifically including a video preview window, a multi-segment timeline, and a text prompt input box associated with the timeline.

[0165] The task scheduling and control module is used to monitor the changes of prompt words in each segment of the timeline in real time. When the video generation process reaches the prompt word change node, it sends a KV-refresh command to the video generation module.

[0166] The video generation module specifically includes: A text encoder is used to convert user-inputted text prompts into vector representations (i.e., text vectors) that the diffusion model can understand. A visual encoder (VAE) is used to convert video frames into vector representations (i.e., visual vectors) that the diffusion model can understand. The KV-refresh module is used to generate target attention parameters and first attention parameters based on text vectors and visual vectors. Diffusion model (DiT) is used to generate video frame sequences based on target attention parameters; The KV cache storage area is a memory area used to store the target attention parameters and the first attention parameters.

[0167] The video generation apparatus of this application is described in detail below. Please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram of one embodiment of the video generation device in this application. The video generation device 1100 includes: The first request processing module 1110 is configured to generate a first video frame sequence corresponding to a first prompt word in response to a first video generation request, wherein the first video generation request carries a first prompt word and first time information, and the first time information is used to indicate a first time interval corresponding to the first video frame sequence. The second request processing module 1120 is used to respond to receiving a second video generation request, wherein the second video generation request carries a second prompt word and second time information, the second time information is used to indicate a second time interval corresponding to the second video frame sequence, and the second time interval is the next time interval adjacent to the first time interval. Attention parameter generation module 1130 is used to generate first attention parameters based on the first video frame sequence and the second prompt word carried in the second video generation request; Attention parameter fusion module 1140 is used to fuse the first attention parameter and the second attention parameter to obtain the target attention parameter, wherein the second attention parameter is determined based on the first cue word and the first video frame sequence; The video frame sequence generation module 1150 is used to generate a second video frame sequence based on the target attention parameters and the first attention parameters, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameters.

[0168] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter fusion module 1140 is further used for: Determine the semantic association parameters between the first and second prompt words; Based on the semantic association parameters, determine the weight values ​​corresponding to the first attention parameter and the second attention parameter; Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0169] Optionally, in the above Figure 11Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter fusion module 1140 is further used for: Based on the learnable matrix, the semantic association parameters are processed to obtain the weight values ​​corresponding to the first value vector; Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0170] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter fusion module 1140 is further used for: If the frame number of the target frame is less than the preset frame number threshold, the transition coefficient is determined based on the frame number of the target frame, where the target frame is the last generated video frame in the second video frame sequence. Based on the transition coefficient, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined. The weight value corresponding to the first attention parameter is positively correlated with the transition coefficient, and the weight value corresponding to the first attention parameter is negatively correlated with the transition coefficient. Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

[0171] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter fusion module 1140 is further used for: The transition coefficients are smoothed using a preset function to obtain the weight values ​​corresponding to the first value vector. The first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficients. Determine the weight value corresponding to the first key vector, where the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the second key vector are used as the weight values ​​corresponding to the second attention parameters.

[0172] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter generation module 1130 is further used for: Generate visual vectors based on the first video frame sequence; Generate a second text vector based on the second prompt word; Linear projection processing is performed on the concatenated visual vector and the second text vector to obtain the first attention parameters.

[0173] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter generation module 1130 is further used for: The first time interval is divided into multiple sub-time intervals; In the first video frame sequence, at least one video frame is extracted from each sub-time interval based on the frame extraction frequency corresponding to each sub-time interval to obtain a reference video frame sequence. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0174] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter generation module 1130 is further used for: Based on the second cue word, identify the target object; For each video frame in the first video frame sequence, target detection is performed on the video frame to obtain the detection result corresponding to the video frame; Based on the detection results corresponding to each video frame, video frames whose detection results contain the target object are extracted from the first video frame sequence to obtain a reference video frame sequence; The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0175] Optionally, in the above Figure 11Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter generation module 1130 is further used for: For each video frame in the first video frame sequence, the area of ​​each target detection box in the video frame is determined based on the coordinates of each target detection box in the detection result corresponding to the video frame. For each video frame in the first video frame sequence, the relevance index corresponding to the video frame is determined based on the area and confidence of each target detection box in the video frame; Based on the relevance index corresponding to each video frame, the K video frames with the largest relevance index or video frames with relevance index exceeding the relevance threshold are extracted from the first video frame sequence to obtain the reference video frame sequence, where K is an integer greater than or equal to 1.

[0176] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the attention parameter generation module 1130 is further used for: Based on the number of historical frames, the minimum total number of frames is determined. The number of historical frames is determined based on the total number of video frames in the first video frame sequence. The number of historical frames is positively correlated with the minimum total number of frames. Based on the minimum total number of frames, N video frames are extracted from the first video frame sequence to obtain a reference video frame sequence, where N is an integer greater than or equal to the minimum total number of frames. The first attention parameters are generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

[0177] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, a request generation module is further included, which is used for: Display the video generation interface; In response to a text input operation on the video generation interface, obtain the second prompt word; In response to a time input operation on the video generation interface, obtain second time information; Based on the second prompt word and the second time information, a second video generation request is generated.

[0178] Optionally, in the above Figure 11 Based on the corresponding embodiments, in another embodiment of the video generation apparatus 1100 provided in this application, the video frame sequence generation module 1150 is further used for: Based on the target attention parameters, generate the first K video frames in the second video frame sequence, where K is an integer greater than or equal to 1.

[0179] Based on the first attention parameter, generate the K+i-th video frame in the second video frame sequence, where i is an integer greater than or equal to 1.

[0180] Figure 12 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 1200 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1222 (e.g., one or more processors) and a memory 1232, and one or more storage media 1230 (e.g., one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and storage media 1230 can be temporary or persistent storage. The program stored in the storage media 1230 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device. Furthermore, the CPU 1222 may be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the computer device 1200.

[0181] Computer device 1200 may also include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input or output interfaces 1258, and / or one or more operating systems 1241, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0182] The steps performed by the computer device in the above embodiments can be based on this Figure 12 The computer device structure shown.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for video generation, characterized in that, The method includes: In response to a first video generation request, a first video frame sequence corresponding to a first prompt word is generated, wherein the first video generation request carries the first prompt word and first time information, and the first time information is used to indicate a first time interval corresponding to the first video frame sequence; Receive a second video generation request, wherein the second video generation request carries a second prompt word and second time information, the second time information being used to indicate a second time interval corresponding to the second video frame sequence, the second time interval being the next time interval adjacent to the first time interval; Generate first attention parameters based on the first video frame sequence and the second cue word carried in the second video generation request; The first attention parameter and the second attention parameter are fused to obtain the target attention parameter, wherein the second attention parameter is determined based on the first cue word and the first video frame sequence; The second video frame sequence is generated based on the target attention parameter and the first attention parameter, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameter.

2. The method according to claim 1, characterized in that, The process of fusing the first attention parameter and the second attention parameter to obtain the target attention parameter includes: Determine the semantic association parameters between the first prompt word and the second prompt word; Based on the semantic association parameters, determine the weight values ​​corresponding to the first attention parameter and the second attention parameter; Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

3. The method according to claim 2, characterized in that, The first attention parameter includes a first value vector and a first key vector; the second attention parameter includes a second value vector and a second key vector. The step of determining the weight values ​​corresponding to the first attention parameter and the second attention parameter based on the semantic association parameters includes: Based on the learnable matrix, the semantic association parameters are processed to obtain the weight values ​​corresponding to the first value vector; Determine the weight value corresponding to the first key vector, wherein the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the weight values ​​corresponding to the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the weight values ​​corresponding to the second key vector are used as the weight values ​​corresponding to the second attention parameters.

4. The method according to claim 1, characterized in that, The process of fusing the first attention parameter and the second attention parameter to obtain the target attention parameter includes: If the frame number of the target frame is less than the preset frame number threshold, then the transition coefficient is determined based on the frame number of the target frame, wherein the target frame is the last generated video frame in the second video frame sequence. Based on the transition coefficient, the weight values ​​corresponding to the first attention parameter and the second attention parameter are determined, wherein the weight value corresponding to the first attention parameter is positively correlated with the transition coefficient, and the weight value corresponding to the first attention parameter is negatively correlated with the transition coefficient. Based on the weight values ​​corresponding to the first attention parameter and the second attention parameter, the first attention parameter and the second attention parameter are fused to obtain the target attention parameter.

5. The method according to claim 4, characterized in that, The first attention parameter includes a first value vector and a first key vector; the second attention parameter includes a second value vector and a second key vector. The step of determining the weight values ​​corresponding to the first attention parameter and the second attention parameter based on the transition coefficient includes: The transition coefficients are smoothed using a preset function to obtain the weight values ​​corresponding to the first value vector. The first derivative of the preset function is greater than or equal to 0, and the second derivative of the preset function is negatively correlated with the transition coefficients. Determine the weight value corresponding to the first key vector, wherein the weight value corresponding to the first key vector is equal to the weight value corresponding to the first value vector; The weight values ​​corresponding to the first value vector and the weight values ​​corresponding to the first key vector are used as the weight values ​​corresponding to the first attention parameters. Determine the weight value corresponding to the second value vector and the weight value corresponding to the second key vector, wherein the sum of the weight value corresponding to the second value vector and the weight value corresponding to the first value vector is 1, and the sum of the weight value corresponding to the second key vector and the weight value corresponding to the first key vector is 1; The weight values ​​corresponding to the second value vector and the weight values ​​corresponding to the second key vector are used as the weight values ​​corresponding to the second attention parameters.

6. The method according to claim 1, characterized in that, The step of generating the first attention parameter based on the first video frame sequence and the second cue word carried in the second video generation request includes: Based on the first video frame sequence, a visual vector is generated; Based on the second prompt word, generate a second text vector; The first attention parameter is obtained by performing linear projection processing on the concatenated visual vector and the second text vector.

7. The method according to claim 1, characterized in that, The step of generating the first attention parameter based on the first video frame sequence and the second cue word carried in the second video generation request includes: The first time interval is divided into multiple sub-time intervals; In the first video frame sequence, at least one video frame is extracted from each sub-time interval based on the frame extraction frequency corresponding to each sub-time interval to obtain a reference video frame sequence. The first attention parameter is generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

8. The method according to claim 1, characterized in that, The step of generating the first attention parameter based on the first video frame sequence and the second cue word carried in the second video generation request includes: Based on the second prompt word, the target object is identified; For each video frame in the first video frame sequence, target detection is performed on the video frame to obtain the detection result corresponding to the video frame; Based on the detection result corresponding to each video frame, video frames whose detection results contain the target object are extracted from the first video frame sequence to obtain a reference video frame sequence; The first attention parameter is generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

9. The method according to claim 8, characterized in that, The detection results include the coordinates of the target detection box and the confidence level of the target detection box; The step of extracting video frames containing the target object from the first video frame sequence based on the detection results corresponding to each video frame to obtain a reference video frame sequence includes: For each video frame in the first video frame sequence, the area of ​​each target detection box in the video frame is determined based on the coordinates of each target detection box in the detection result corresponding to the video frame. For each video frame in the first video frame sequence, a relevance index corresponding to the video frame is determined based on the area and confidence of each target detection box in the video frame. Based on the relevance index corresponding to each video frame, the K video frames with the largest relevance index or video frames with a relevance index exceeding the relevance threshold are extracted from the first video frame sequence to obtain the reference video frame sequence, where K is an integer greater than or equal to 1.

10. The method according to claim 1, characterized in that, The step of generating the first attention parameter based on the first video frame sequence and the second cue word carried in the second video generation request includes: Based on the number of historical frames, a minimum total number of frames is determined, wherein the number of historical frames is determined based on the total number of video frames in the first video frame sequence, and the number of historical frames is positively correlated with the minimum total number of frames; Based on the minimum total number of frames, N video frames are extracted from the first video frame sequence to obtain a reference video frame sequence, where N is an integer greater than or equal to the minimum total number of frames. The first attention parameter is generated based on the reference video frame sequence and the second cue word carried in the second video generation request.

11. The method according to claim 1, characterized in that, The method further includes: Display the video generation interface; In response to a text input operation on the video generation interface, the second prompt word is obtained; In response to a time input operation on the video generation interface, the second time information is obtained; Based on the second prompt word and the second time information, the second video generation request is generated.

12. The method according to claim 1, characterized in that, The step of generating the second video frame sequence based on the target attention parameters and the first attention parameters includes: Based on the target attention parameters, the first K video frames in the second video frame sequence are generated, where K is an integer greater than or equal to 1; Based on the first attention parameter, the K+i-th video frame in the second video frame sequence is generated, where i is an integer greater than or equal to 1.

13. A video generation apparatus, characterized in that, The device includes: The first request processing module is configured to generate a first video frame sequence corresponding to a first prompt word in response to a first video generation request, wherein the first video generation request carries the first prompt word and first time information, and the first time information is used to indicate a first time interval corresponding to the first video frame sequence. The second request processing module is used to respond to receiving a second video generation request, wherein the second video generation request carries a second prompt word and second time information, the second time information is used to indicate a second time interval corresponding to the second video frame sequence, and the second time interval is the next time interval adjacent to the first time interval; The attention parameter generation module is used to generate a first attention parameter based on the first video frame sequence and the second prompt word carried in the second video generation request; An attention parameter fusion module is used to fuse the first attention parameter and the second attention parameter to obtain the target attention parameter, wherein the second attention parameter is determined based on the first prompt word and the first video frame sequence; A video frame sequence generation module is configured to generate a second video frame sequence based on the target attention parameter and the first attention parameter, wherein at least one video frame in the second video frame sequence that is adjacent to the first video frame sequence is generated based on the target attention parameter.

14. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is configured to execute a program in the memory, including performing the method as described in any one of claims 1 to 12; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

15. A computer-readable storage medium comprising instructions, characterized in that, When it is run on a computer, it causes the computer to perform the steps of the method as described in any one of claims 1 to 12.

16. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 12.

Images