A method, system, electronic device, and computer-readable medium for generating multimodal long videos based on read-write memory tokens and boundary self-consistency constraints.

Abstract

This invention discloses a method, system, electronic device, and computer-readable medium for generating multimodal long videos based on readable and writable memory tokens and boundary self-consistency constraints. The method includes: encoding the input multimodal information and obtaining a unified conditional representation through gating fusion; dividing the target video into multiple adjacent segments with overlapping regions; establishing a cross-segment shared memory token pool, reading previous memories as conditional constraints when generating the current segment, and extracting features of the current segment to update the memory pool after generation; generating each segment sequentially based on a latent space diffusion model, using unified conditions and memory tokens as constraints; applying latent space, perceptual appearance, and motion trend consistency constraints to the overlapping regions of adjacent segments during training to optimize model parameters; and performing weighted fusion of pixels in the overlapping areas during inference to output a complete long video. Through memory read / write and boundary self-consistency constraints, the consistency of identity and scene during long-term generation is effectively maintained, and segment splicing flicker is eliminated.

Description

Technical Field

[0001] This invention relates to the field of long video generation technology, and in particular to a multimodal long video generation method, system, electronic device, and computer-readable medium based on a readable and writable memory token and boundary self-consistency constraints. Background Technology

[0002] Existing multimodal video generation technologies mainly refer to the generation of temporally continuous, spatially clear, and semantically aligned video sequences under input conditions such as text, images, audio, motion, or structural control. Current mainstream technical approaches mainly include diffusion-based video generation, latent space video generation, discrete token video generation, keyframe or segmented long video generation, and controllable video generation.

[0003] Among them, the diffusion-based video generation scheme usually uses a diffusion model as the core generation backbone to gradually denoise and generate video in pixel space or latent space; the latent space video generation scheme first compresses the video into the latent space through a video encoder, and then performs diffusion modeling in the latent space to reduce the amount of computation and video memory usage.

[0004] Discrete token video generation schemes first represent the video as a discrete spatiotemporal token sequence using a video tokenizer, and then utilize Transformer, autoregressive model, or mask modeling model to generate long sequences, thereby improving long-term modeling capabilities. For long video tasks, existing technologies often divide the target video into multiple segments, or generate keyframes first, and then perform intermediate frame completion, interpolation, or super-resolution reconstruction to alleviate the computational pressure caused by directly generating long-term video sequences.

[0005] In addition, to improve the controllability of video content, existing solutions also introduce structural control signals such as pose, depth, sketch, trajectory, and keyframes to impose constraints on the generation process.

[0006] Although existing technologies are capable of generating shorter video clips, they still have significant limitations in long-term, multimodal, and highly controllable scenarios.

[0007] First, long-term consistency is poor: as video length increases, problems such as subject identity drift, sudden disappearance or proliferation of objects, changes in clothing and scene style, and unnatural shot transitions are likely to occur.

[0008] Second, the segment splicing boundaries are discontinuous: Existing long video generation technologies usually adopt segment generation, but there is a lack of stable information transmission mechanism between adjacent segments. Therefore, in the overlapping area of ​​segments or at the segment boundary, appearance jumps, action breaks and semantic discontinuities are likely to occur.

[0009] Third, the generation cost is high and the efficiency is low: diffusion methods usually require dozens or even hundreds of sampling steps, which greatly increases the computational cost for long video tasks, and the inference latency is high, making it difficult to balance high resolution and long duration.

[0010] Fourth, the multimodal control signals are not fully utilized: existing methods often can only handle single or a small number of control conditions, and there is a lack of a unified information fusion mechanism between text, audio, and structural control, resulting in unstable control effects. Summary of the Invention

[0011] According to a first aspect of the present invention, a method for generating multimodal long videos based on a read-write memory token and boundary self-consistency constraints is provided, comprising the following steps: The system acquires input multimodal information, which includes at least text prompts and optionally at least one of reference images, audio sequences, and sparse structure control signals; the multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation; The target video is acquired and divided into multiple adjacent video segments along the timeline, with a preset number of overlapping frames between adjacent video segments. Establish a shared memory token pool across segments, and when generating each video segment, read the updated memory token pool of the previous video segment as the generation condition for the current video segment; after the current video segment is generated, extract the features of the current video segment to update the memory token pool. Using a unified conditional representation and the retrieved memory token pool as constraints, the latent space video diffusion model is used to generate each video segment sequentially. Calculate the boundary self-consistency loss of adjacent video segments within the overlapping region to constrain the parameters of the latent space video diffusion model; The pixels in the overlapping area of ​​adjacent video segments are weighted and fused to output the complete target long video.

[0012] Furthermore, the multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation. Specifically: Text prompts, reference images, audio sequences, and sparse structure control signals are encoded using text encoders, image encoders, audio encoders, and structure control encoders respectively to obtain corresponding feature representations; The weight coefficients corresponding to each mode are calculated through a gating fusion mechanism, and the sum of the weight coefficients is 1. The feature representations of multimodal information are weighted and fused with their corresponding weight coefficients to obtain a unified conditional representation.

[0013] Furthermore, the number of video segments N is calculated according to the following formula: ; Where F is the total number of frames in the target video, L is the number of frames in each video segment, and K is the number of frames in the overlapping region, and 0 <K<L。

[0014] Furthermore, a shared memory token pool is established across segments. When generating each video segment, the updated memory token pool of the previous video segment is read as the generation condition for the current video segment. After the current video segment is generated, the features of the current video segment are extracted to update the memory token pool. Specifically: By passing the unified conditional representation through a projection network, an initial memory token pool is obtained. ; When generating the nth video segment, the memory token pool of the previous segment will be used. The unified conditional representation is input into the spatiotemporal attention module of the latent space video diffusion model so that the generation of the current video segment inherits the contextual features of the previous segment. Feature extraction and spatiotemporal pooling are performed on the generated nth video segment to obtain a summary vector, and a memory retention coefficient λ is used to pool the preceding memory tokens. The updated memory token pool is obtained by performing a weighted update on the digest vector. .

[0015] Furthermore, the memory token pool includes an identity sub-pool, a scene sub-pool, and a motion sub-pool, which are used to transmit the subject's identity, scene style, and motion status information, respectively.

[0016] Furthermore, the boundary self-consistency loss is expressed as: ; in, and These represent the latent space representations of two adjacent segments within the overlapping interval; Operators for extracting optical flow or motion features; , , The preset hyperparameter weights, In order to perceive loss, .

[0017] Furthermore, a comparative constraint is introduced between the main features of the current video segment and the identity memory vector in the prior memory token pool.

[0018] According to a second aspect of the present invention, a multimodal long video generation system based on a read-write memory token and boundary self-consistency constraints is provided, comprising: A generation module is constructed to acquire input multimodal information, which includes at least text prompts and optionally at least one of reference images, audio sequences, and sparse structure control signals; the multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation; The video segmentation module is used to acquire the target video and divide the target video into multiple adjacent video segments along the time axis, wherein there is a preset number of overlapping areas between adjacent video segments; The memory read / write module is used to establish a shared memory token pool across segments. When generating each video segment, it reads the updated memory token pool of the previous video segment as the generation condition for the current video segment. After the current video segment is generated, it extracts the features of the current video segment to update the memory token pool. The diffusion generation module is used to generate video segments sequentially using the latent space video diffusion model, with unified condition representation and the read memory token pool as constraints. The boundary constraint module is used to calculate the boundary self-consistency loss of adjacent video segments in the overlapping area, so as to constrain the parameters of the latent space video diffusion model. The fusion output module is used to perform weighted fusion of pixels in the overlapping area of ​​adjacent video segments to output the complete target long video.

[0019] According to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor executes the computer program to perform a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to the first aspect.

[0020] According to a fourth aspect of the present invention, a computer-readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to perform a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to the first aspect.

[0021] A multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to an embodiment of the present invention has the following beneficial effects: 1. Significantly improves the consistency of identity and scene in long-sequence generation. This invention establishes a cross-segment shared read-write memory token pool and sets up three independent memory sub-pools for identity, scene, and motion, achieving explicit transmission of long-range context. Compared with existing schemes that only calculate self-attention within a single frame, this invention dynamically updates the identity features and scene state of the currently generated segment to the memory pool through a write-back mechanism, enabling subsequent segment generation to read previous memories through cross-attention. This mechanism effectively suppresses the identity drift and scene style abruptness problems commonly found in long video generation, and is particularly suitable for video generation tasks exceeding 10 seconds in length and containing complex subject motion.

[0022] 2. To fundamentally solve the problems of flickering and motion breakage at segmented splicing points, this invention designs a multi-dimensional boundary self-consistency constraint mechanism to address the unavoidable boundary seam imperfections in segmented generation. This mechanism enforces consistency in three aspects during the training phase: latent space values, perceptual appearance features, and motion optical flow trends in overlapping areas of adjacent segments. Compared to existing methods that rely solely on post-processing fusion, this invention optimizes the smoothness of segmented boundaries from the root of model training, significantly reducing screen flickering, subject jumps, and motion breakage at splicing points during the inference phase.

[0023] 3. To achieve unified control and flexible expansion of multimodal signals, this invention constructs a unified multimodal conditional coding framework through a gating fusion mechanism. This framework can adaptively calculate the weight coefficients of text, image, audio, and structural control signals, mapping them to the same conditional space. This mechanism not only ensures that the generated video strictly follows the text semantics and reference appearance, but also accurately responds to audio rhythm and sparse structural constraints (such as keypoint trajectories), overcoming the semantic conflicts or control imbalances that easily occur in existing technologies when multiple signal sources are superimposed.

[0024] 4. This invention possesses excellent system compatibility and ease of integration. It employs a modular design, where the latent space diffusion generation backbone, memory read / write module, and boundary constraint module are all independent of the specific underlying network architecture. This solution requires no modification to the basic structure of existing single-segment video diffusion models and can be used to overlay long-term memory and consistency constraint functions in a plug-and-play manner, facilitating rapid integration and deployment within existing video generation systems.

[0025] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0026] Figure 1 This is a flowchart of a multimodal long video generation method based on a readable and writable memory token and boundary self-consistency constraints according to an embodiment of the present invention.

[0027] Figure 2 This is a structural diagram of a multimodal long video generation system based on a readable and writable memory token and boundary self-consistency constraints according to an embodiment of the present invention.

[0028] Figure 3 This is a structural diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0029] The preferred embodiment of the first aspect of the present invention will be described in detail below with reference to the accompanying drawings, further illustrating the present invention.

[0030] First, combine Figure 1 This paper describes a method for generating multimodal long videos based on a read-write memory token and boundary self-consistency constraints, according to an embodiment of the present invention. This method has a wide range of applications.

[0031] The multimodal long video generation method provided by this invention includes six stages: multimodal conditional coding, long video segmentation scheduling, cross-segment memory token reading and writing, segmented latent space diffusion generation, boundary self-consistency constraint optimization, and overlapping region fusion output.

[0032] For ease of explanation, let the target video be... ,in Indicates the first Frame image, This indicates the total number of frames in the target video.

[0033] like Figure 1 As shown in the figure, a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to an embodiment of the present invention specifically includes the following steps: like Figure 1 As shown, in S1, the multimodal conditional coding step is as follows: The input multimodal information is obtained, which includes at least a text prompt (…). (and optionally include reference images) Audio sequences and sparse structure control signals At least one of the following: Multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation. It should be noted that text encoders are used separately. Image encoder Audio encoder and structural control encoder The text prompts, reference images, audio sequences, and sparse structure control signals are encoded to obtain the corresponding feature representations; the feature representations are as follows: .

[0034] To achieve unified injection of multimodal information, this invention introduces a gating fusion mechanism to calculate the weights of each modality. and satisfy The unified condition C is obtained by the following formula: in, This represents a weighted fusion after feature concatenation or linear mapping. The above formula maps text semantics, appearance references, audio rhythm, and structural control information to the same conditional space, providing unified constraints for subsequent video generation.

[0035] like Figure 1 As shown, in S2, the long video segmentation scheduling step involves: acquiring the target video and dividing it into multiple adjacent video segments along the timeline, where adjacent video segments overlap by a preset number of frames. This step addresses the problem of excessive computation and difficulty in maintaining consistency when directly generating long time-series videos. Specifically, it involves dividing the target video along the timeline into... Each of the adjacent segments contains 10 adjacent segments. Frames, with preservation between adjacent segments Frame overlap region, where Number of segments Determine by the following formula: No. Video segments Represented as: In one specific embodiment, take frame, The video frame rate is 24 fps; when the target video duration is 20 seconds, the total number of frames is... This can be divided into 19 overlapping segments.

[0036] like Figure 1 As shown, in S3, the cross-segment memory token read / write steps are as follows: A cross-segment shared memory token pool is established, and when generating each video segment, the updated memory token pool of the previous video segment is read as the generation condition for the current video segment; after the current video segment is generated, the features of the current video segment are extracted to update the memory token pool. This step is used to continuously transmit subject identity, scene style, and motion state information between different segments. Specifically, it involves setting up a cross-segment shared memory token pool. The initial memory is represented by a unified conditional representation C through a projection network. get: in, J represents the number of memory slots. This represents a single token dimension. The memory token pool includes an identity sub-pool. Scene sub-pool and sports sub-pool In this embodiment, we take... .

[0037] When generating the nth segment, the memory token pool will be used. The unified conditional representation C is input into the video to generate the backbone. Conditional reading is performed in the l-th layer spatiotemporal attention module, and the updated features are then processed. for: in, Indicates the first Hidden features of layers This represents the cross-attention operation. This indicates cascading. Using the above formula, the current segment can explicitly inherit the identity, style, and motion context from the preceding segment during generation.

[0038] After the current segment generation is completed, extract the summary vector from the intermediate features of the generated segments or the decoded video features. And write it back to the memory token pool as follows: in, This indicates that the nth video segment has been generated. This represents a video feature extraction network. This indicates a spacetime pooling operation. For a learnable and updateable matrix, This is the memory retention coefficient. In this embodiment, it is taken as... The above formula ensures that memory information retains historically stable characteristics while absorbing the new state of the current segment when it is propagated across segments.

[0039] like Figure 1 As shown, in S4, the segmented latent space diffusion generation steps are as follows: using the unified condition representation and the read memory token pool as constraints, the latent space video diffusion model is used to sequentially generate each video segment. Specifically, firstly, the video encoder is used... Segmenting the real video Compressing to the latent space yields latent variables. During training, noise is sampled from a standard Gaussian distribution. And add noise to the latent variables at time step t: Subsequently, the noise reduction network In the unified conditional representation C and the memory token pool Predicting noise under common constraints yields the diffusion training objective: in, The cumulative noise coefficient is calculated for the diffusion process. After training using the above formula, the model can simultaneously satisfy semantic alignment and cross-segment consistency requirements when generating each segment.

[0040] like Figure 1 As shown, in S5, the boundary self-consistency constraint optimization step involves calculating the boundary self-consistency loss of adjacent video segments within the overlapping region to constrain the parameters of the latent space video diffusion model. This step primarily addresses the issues of flickering, subject jumps, and motion breaks that easily occur at segment boundaries in existing segmentation generation methods. Specifically, it involves: for adjacent segments... and Overlapping areas Construct a boundary self-consistency loss. Let... and These represent the latent space representations of two adjacent segments within the overlapping interval. If the operator represents optical flow or motion feature extraction, then the boundary loss is defined as: The first constraint constrains the continuity of the latent space, the second constrains the consistency of perceived appearance, and the third constrains the consistency of motion trends. In this embodiment, the preferred constraint is... .

[0041] Furthermore, to suppress identity drift, a comparison constraint can be introduced between the remembered identity token and the current segment's subject characteristics. Let... For identity memory aggregation vector, Given the current segment's main region features, the identity preservation loss can be written as: in, Represents cosine similarity. This represents the temperature coefficient.

[0042] like Figure 1 As shown, in S6, the overlapping region fusion output step involves weighted fusion of pixels in the overlapping region of adjacent video segments to output the complete target long video. It should be noted that the overall training objective of this invention is composed of diffusion denoising loss, boundary self-consistency loss, and identity preservation loss. In this embodiment, the following is selected: After the model training is complete, the inference phase generates data sequentially according to the segment order. And a linear fusion strategy is used to complete the splicing in the overlapping area. If the outputs of the preceding and following segments corresponding to the r-th frame in the overlapping area are respectively and The fusion result is: This formula ensures smooth pixel and motion transitions during segment splicing, ultimately resulting in a complete long video output. .

[0043] As described above, the multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to an embodiment of the present invention has the following beneficial effects: 1. Significantly improves the consistency of identity and scene in long-sequence generation. This invention establishes a cross-segment shared read-write memory token pool and sets up three independent memory sub-pools for identity, scene, and motion, achieving explicit transmission of long-range context. Compared with existing schemes that only calculate self-attention within a single frame, this invention dynamically updates the identity features and scene state of the currently generated segment to the memory pool through a write-back mechanism, enabling subsequent segment generation to read previous memories through cross-attention. This mechanism effectively suppresses the identity drift and scene style abruptness problems commonly found in long video generation, and is particularly suitable for video generation tasks exceeding 10 seconds in length and containing complex subject motion.

[0044] 2. To fundamentally solve the problems of flickering and motion breakage at segmented splicing points, this invention designs a multi-dimensional boundary self-consistency constraint mechanism to address the unavoidable boundary seam imperfections in segmented generation. This mechanism enforces consistency in three aspects during the training phase: latent space values, perceptual appearance features, and motion optical flow trends in overlapping areas of adjacent segments. Compared to existing methods that rely solely on post-processing fusion, this invention optimizes the smoothness of segmented boundaries from the root of model training, significantly reducing screen flickering, subject jumps, and motion breakage at splicing points during the inference phase.

[0045] 3. To achieve unified control and flexible expansion of multimodal signals, this invention constructs a unified multimodal conditional coding framework through a gating fusion mechanism. This framework can adaptively calculate the weight coefficients of text, image, audio, and structural control signals, mapping them to the same conditional space. This mechanism not only ensures that the generated video strictly follows the text semantics and reference appearance, but also accurately responds to audio rhythm and sparse structural constraints (such as keypoint trajectories), overcoming the semantic conflicts or control imbalances that easily occur in existing technologies when multiple signal sources are superimposed.

[0046] 4. This invention possesses excellent system compatibility and ease of integration. It employs a modular design, where the latent space diffusion generation backbone, memory read / write module, and boundary constraint module are all independent of the specific underlying network architecture. This solution requires no modification to the basic structure of existing single-segment video diffusion models and can be used to overlay long-term memory and consistency constraint functions in a plug-and-play manner, facilitating rapid integration and deployment within existing video generation systems.

[0047] The above combined with the appendix Figure 1 This paper describes a method for generating multimodal long videos based on a read-write memory token and boundary self-consistency constraints, according to an embodiment of the present invention. Furthermore, the present invention can also be applied to a multimodal long video generation system based on a read-write memory token and boundary self-consistency constraints.

[0048] like Figure 2 As shown, according to a second aspect of the present invention, a multimodal long video generation system based on a read-write memory token and boundary self-consistency constraints is provided, comprising: A generation module 100 is constructed to acquire input multimodal information, which includes at least text prompts and optionally at least one of reference images, audio sequences, and sparse structure control signals; the multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation; The video segmentation module 200 is used to acquire the target video and divide the target video into multiple adjacent video segments along the time axis, wherein there is an overlap area with a preset number of frames between adjacent video segments. The memory read / write module 300 is used to establish a cross-segment shared memory token pool. When generating each video segment, it reads the updated memory token pool of the previous video segment as the generation condition for the current video segment. After the current video segment is generated, it extracts the features of the current video segment to update the memory token pool. The diffusion generation module 400 is used to generate video segments sequentially using the latent space video diffusion model, with unified condition representation and the read memory token pool as constraints. Boundary constraint module 500 is used to calculate the boundary self-consistency loss of adjacent video segments in the overlapping area, so as to constrain the parameters of the latent space video diffusion model; The fusion output module 600 is used to perform weighted fusion of pixels in the overlapping area of ​​adjacent video segments to output a complete target long video.

[0049] The above combined with the appendix Figure 2This paper describes a multimodal long video generation system based on a read-write memory token and boundary self-consistency constraints according to an embodiment of the present invention. It expresses the system scheme of the method, so the details are not repeated here, as they are consistent with the method. Furthermore, the present invention can also be applied to an electronic device.

[0050] like Figure 3 As shown, according to a third aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor executes the computer program to perform a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to the first aspect.

[0051] According to a fourth aspect of the present invention, a computer-readable medium having processor-executable non-volatile program code is provided, the program code causing the processor to perform a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints according to the first aspect.

[0052] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of a computer program from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the ASIC can reside within a device. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc. The present invention also provides a program product comprising executable instructions stored in the readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions, causing the device to implement the multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints provided in the various embodiments described above. In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0053] It should be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for multi-modal long video generation based on read-write memory Token and boundary self-consistency constraints, characterized in that, It includes the following steps: The system acquires input multimodal information, which includes at least text prompts and optionally at least one of reference images, audio sequences, and sparse structure control signals; it encodes the multimodal information separately and calculates the weights of the multimodal information through a gating fusion mechanism to generate a unified conditional representation; The target video is acquired and divided into multiple adjacent video segments along the timeline, with a preset number of overlapping frames between adjacent video segments. Establish a shared memory token pool across segments, and when generating each video segment, read the updated memory token pool of the previous video segment as the generation condition for the current video segment. After the current video segment is generated, the features of the current video segment are extracted to update the memory token pool; Using the unified condition representation and the memory token pool read as constraints, the video segments are generated sequentially using the latent space video diffusion model; Calculate the boundary self-consistency loss of adjacent video segments within the overlapping region to constrain the parameters of the latent space video diffusion model; The pixels of adjacent video segments within the overlapping area are weighted and fused to output a complete target long video.

2. The method of claim 1, wherein the method is based on a read-write memory token and boundary self-consistency constraints for multi-modal long video generation. The specific steps of encoding the multimodal information separately and calculating the weights of the multimodal information through a gating fusion mechanism to generate a unified conditional representation are as follows: The text prompt, reference image, audio sequence, and sparse structure control signal are encoded using a text encoder, image encoder, audio encoder, and structure control encoder respectively to obtain corresponding feature representations; The weight coefficients corresponding to each mode are calculated through the gating fusion mechanism, and the sum of the weight coefficients is 1. The unified conditional representation is obtained by weighting and fusing the feature representations of multimodal information with their corresponding weight coefficients.

3. The multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in claim 1, characterized in that... The number N of video segments is calculated according to the following formula: ； Where F is the total number of frames in the target video, L is the number of frames in each video segment, and K is the number of frames in the overlapping region, and 0 <K<L。 4. The multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in claim 1, characterized in that, The establishment of a cross-segment shared memory token pool, and the reading of the updated memory token pool of the previous video segment as the generation condition for the current video segment when generating each video segment; and the extraction of features of the current video segment to update the memory token pool after the current video segment is generated, specifically involves: The unified condition representation is passed through a projection network to obtain the initial memory token pool. ; When generating the nth video segment, the memory token pool of the previous segment will be used. The unified condition is input into the spatiotemporal attention module of the latent space video diffusion model so that the generation of the current video segment inherits the contextual features of the previous segment. Feature extraction and spatiotemporal pooling are performed on the generated nth video segment to obtain a summary vector, and a memory retention coefficient λ is used to pool the preceding memory tokens. The updated memory token pool is obtained by weighting and updating the digest vector. .

5. The multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in claim 4, characterized in that, The memory token pool includes an identity sub-pool, a scene sub-pool, and a motion sub-pool, which are used to transmit the subject's identity, scene style, and motion status information, respectively.

6. The multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in claim 1, characterized in that, The boundary self-consistency loss is expressed as: ； in, and These represent the latent space representations of two adjacent segments within the overlapping interval; Operators for extracting optical flow or motion features; , , The preset hyperparameter weights, In order to perceive loss, .

7. The multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in claim 1 or 6, characterized in that, A comparative constraint is introduced between the main features of the current video segment and the identity memory vector in the previous memory token pool.

8. A multimodal long video generation system based on a read-write memory token and boundary self-consistency constraints, characterized in that, Include: A generation module is constructed to acquire input multimodal information, which includes at least text prompts and optionally at least one of reference images, audio sequences, and sparse structure control signals; the multimodal information is encoded separately, and the weights of the multimodal information are calculated through a gating fusion mechanism to generate a unified conditional representation; The video segmentation module is used to acquire the target video and divide the target video into multiple adjacent video segments along the time axis, wherein there is a preset number of overlapping areas between adjacent video segments; The memory read / write module is used to establish a cross-segment shared memory token pool, and when generating each video segment, it reads the updated memory token pool of the previous video segment as the generation condition of the current video segment; after the current video segment is generated, it extracts the features of the current video segment to update the memory token pool. The diffusion generation module is used to generate each video segment sequentially using the latent space video diffusion model, with the unified condition representation and the read memory token pool as constraints. The boundary constraint module is used to calculate the boundary self-consistency loss of adjacent video segments within the overlapping region, so as to constrain the parameters of the latent space video diffusion model. The fusion output module is used to perform weighted fusion of pixels in the overlapping area of ​​adjacent video segments to output a complete target long video.

9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor executes the computer program to perform a multimodal long video generation method based on a read-write memory token and boundary self-consistency constraints as described in any one of claims 1 to 7.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the multimodal long video generation method based on readable and writable memory tokens and boundary self-consistency constraints as described in any one of claims 1-7.

Images