A deep learning-based video generation method and system

Abstract

This invention discloses a deep learning-based video generation method and system, comprising the following steps: acquiring input content including text instructions and reference images, generating a scene layered relationship graph and visual patch tokens; constructing persistent relationship features based on the scene layered relationship graph and local relationship fragments and encoding them into a persistent relationship token sequence; inputting the noisy persistent relationship token sequence into a topology-aware DiT model to generate a dynamic topology graph sequence; generating a depth map sequence and an object index map sequence based on the dynamic topology graph sequence, and establishing an object continuity directory; inputting the noisy visual patch tokens and the object continuity directory into a geometry-guided DiT model to obtain a video latent space feature sequence; performing spatiotemporal consistency decoding, and outputting the target video stream. This invention improves relationship consistency, occlusion recovery capability, and spatiotemporal continuity in the video generation process.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and video generation technology, and in particular to a video generation method and system based on deep learning. Background Technology

[0002] With the continuous development of diffusion models, Transformer architecture, and multimodal conditional generation techniques, video generation technology based on text instructions and reference images has received widespread attention. Existing video generation methods mainly rely on end-to-end generation within a unified latent space, or on applying overall constraints to the video content through textual or image conditions. However, these methods commonly suffer from the following problems in practical applications: Existing methods typically treat objects, relationships, and motion processes in a scene as unified visual features for hybrid modeling, lacking explicit expression of scene topological relationships. This makes it difficult to maintain stable connections, occlusion relationships, and motion associations between objects in multi-object interaction scenarios, resulting in structural drift and relationship misalignment in the generated results. For long-term video generation tasks, existing technologies often use frame-by-frame independent processing or short-term context aggregation for local relationship information, lacking cross-segment relationship propagation and continuous expression mechanisms. This leads to insufficient continuity of object relationships in the temporal dimension, easily causing action jumps and interaction distortions. Furthermore, in scenarios where objects are occluded, leave the field of view, and then reappear, existing methods typically lack a mechanism for continuous registration and effective retrieval of the target object's historical stable appearance information. This results in texture inconsistencies, identity drift, and temporal flickering in the reappeared area, thus affecting the geometric consistency and visual coherence of the generated video.

[0003] Therefore, how to provide a video generation method and system based on deep learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] One objective of this invention is to propose a video generation method and system based on deep learning. This invention fully utilizes layered topological scene decomposition, persistent relation modeling, topology-aware DiT model, geometry-guided DiT model, and VAE spatiotemporal consistency decoding technology, and describes in detail the video generation process oriented towards text instructions and reference images. It can achieve high-quality generation of target video streams in complex scenes and has the advantages of strong relation consistency, good occlusion recovery effect, high spatiotemporal continuity, and strong generation stability.

[0005] A deep learning-based video generation method according to an embodiment of the present invention includes the following steps: Step 1: Obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; Step 2: Perform local state registration, consistency correction, and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence features, and encode the relationship persistence features into a relationship persistence token sequence; Step 3: Input the noisy relation persistence token sequence into the topology-aware DiT model. The topology-aware DiT model introduces a graph node embedding layer, topology position encoding, topology sparse attention and motion parameter decoder to generate a dynamic topology graph sequence. Step 4: Perform projective geometric projection and visibility layering calculations on the dynamic topology graph sequence to generate a depth map sequence and an object index map sequence; Step 5: Perform persistent occlusion cataloging based on the depth map sequence and object index map sequence, continuously register the visible state range and stable appearance fragments of the target object, and establish an object continuity catalog; Step 6: Input the noisy visual Patch Token and object continuity catalog into the geometry-guided DiT model, which introduces a geometric conditional encoder and geometric cross attention to obtain the video latent space feature sequence; Step 7: Input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding, and output the target video stream.

[0006] Optionally, step one specifically includes: The text instructions are parsed syntactically to extract object terms representing scene objects and relation terms representing the relationships between objects. Based on the syntactic parsing results, the attributive and quantifier words modifying the same object term are merged to generate entity primitives. Relation terms that have dependency connections with two object terms are identified as relation primitives. The reference image is divided into regions to obtain multiple superpixel regions. The Canny edge detection operator is used to extract the boundary information of each superpixel region. Image regions with closed boundaries and connected component areas greater than a preset pixel threshold are identified as object candidate regions. Common boundary regions, overlapping boundary regions, and adjacent regions located between adjacent object candidate regions are identified as relation candidate regions. The object candidate regions are divided into object regions, background regions, and relation regions. Alignment matching is performed between object terms in entity primitives and object regions, and association mapping is performed between relation terms in relation primitives and relation regions. Object nodes are generated from matched object regions, and relation edges are generated from mapped relation regions, thus generating a scene hierarchical relation graph. The reference image is divided and encoded into image blocks to generate a visual PatchToken that corresponds to each image block.

[0007] Optionally, step two specifically includes: Traverse each object node in the scene hierarchical relationship graph, select all relationship edges that are directly connected to the current object node, read the relationship region corresponding to each relationship edge, and combine the current object node, relationship edge, and relationship region into a local relationship fragment. For each local relation segment, read the object identifier, object category, and region location of the object node; for each local relation segment, read the relation category, source object identifier, and target object identifier of the relation edge; and write the results into the local state record. The results of each local state record are classified according to the object identifier. Local state record results with the same object identifier are divided into the same comparison group, and consistency correction is performed within each comparison group. For each comparison group after consistency correction, cross-fragment relationship propagation is performed according to the connection relationship between object nodes. The relationship category, target object identifier, and relationship region location in the current local state record result are written into the local state record result adjacent to the current object node, and the corresponding content in the adjacent local state record result is overwritten with the written relationship category, target object identifier, and relationship region location. After completing the cross-segment relationship propagation, the local state record results are sequentially concatenated according to the order of object identifier, relationship category, region location, and target object identifier to generate relationship persistence feature items; The persistent features of each relationship are sorted according to the connection order of the object nodes in the scene hierarchical relationship graph, and the sorted persistent features are input into the sequence encoding layer and encoded into a persistent token sequence.

[0008] Optionally, step three specifically includes: According to the perturbation intensity corresponding to the current diffusion time step, a random noise component is superimposed on the relationship persistence token sequence; The topology-aware DiT model includes an input processing layer, a core backbone layer, and an output processing layer: The input processing layer includes a graph node embedding layer, a topology location encoding unit, and a time step embedding unit: the graph node embedding layer maps the relationship persistence token sequence to graph node tokens; the topology location encoding unit constructs a graph Laplacian matrix based on the scene hierarchical relationship graph, performs feature decomposition on the graph Laplacian matrix, selects the obtained feature vector as the relative position representation of the node in the topology, and superimposes the relative position representation into the corresponding graph node token; the time step embedding unit inputs the current diffusion time step into the multilayer perceptron and outputs a time step condition vector. The core backbone layer consists of multiple Transformer Blocks arranged in series. Each Transformer Block includes an adaptive layer normalization unit, a topological sparse attention unit, and a point-by-point feedforward network unit. The adaptive layer normalization unit modulates the normalization result of the graph node tokens input to the current layer according to the time step condition vector. The topological sparse attention unit introduces a topological binary mask matrix. The topological binary mask matrix is ​​constructed based on the node connection relationship in the scene layer relationship graph. The topological binary mask matrix is ​​only allowed to participate in the attention weight calculation when the corresponding node pairs have a connection relationship. The point-by-point feedforward network unit performs nonlinear dimensionality upscaling and projection on the node features output by the topological sparse attention unit to obtain the feature update result of the current Transformer Block. The output processing layer includes a motion parameter decoder. The motion parameter decoder receives the node hidden features output by the last Transformer Block and converts the node hidden features into velocity parameters, displacement change parameters, and relation change parameters of the object nodes and their edges, respectively. Based on the velocity parameters and displacement change parameters of the object nodes, the spatiotemporal position results of the object nodes at each time moment are generated. Based on the relation change parameters of the relation edges, the connection results of the relation edges and the position results of the relation regions at each time moment are generated. The spatiotemporal position results of the object nodes, the connection results of the relation edges, and the position results of the relation regions are combined in chronological order to form a dynamic topology graph sequence.

[0009] Optionally, step four specifically includes: The focal length parameters, principal point position parameters, and camera pose parameters are read according to the preset virtual camera parameters, and a geometric projection matrix is ​​constructed based on the focal length parameters, principal point position parameters, and camera pose parameters. Read the three-dimensional position results of the object nodes at each time point in the dynamic topology graph sequence in chronological order, and read the horizontal coordinate, vertical coordinate and depth coordinate of the three-dimensional position result of each object node. Input the horizontal, vertical, and depth coordinates of each object node into the geometric projection matrix, perform coordinate transformation and projection conversion, and obtain the projected coordinates of each object node in the two-dimensional image plane. Centered on the projection coordinates of each object node in the two-dimensional image plane, and based on the spatial distribution range of the object nodes at the current moment, the object projection area of ​​each object in the two-dimensional image plane is determined. The projection regions of each object in the two-dimensional image plane are traversed pixel by pixel, and the corresponding depth coordinates of multiple object projection regions located at the same pixel position are read. Compare the depth coordinates of the projected areas of each object at the same pixel position, and determine the projected area of ​​the object with the smaller depth coordinate as the front area and the projected area of ​​the object with the larger depth coordinate as the back area. Repeatedly perform depth coordinate comparison on all pixel positions in the two-dimensional image plane to obtain the front and back region distributions of the projection areas of each object in the entire image at the current moment. The Z-Buffer algorithm is applied to the front and back region distributions obtained by visibility layering calculation. At each pixel position, the depth value corresponding to the front region is retained, and the occluded part of the back region at the pixel position is deleted. Write the depth values ​​at each pixel position at the current time into the corresponding two-dimensional image position to generate the depth map at the current time. Write the object identifiers preserved at each pixel position at the current moment into the corresponding two-dimensional image position to generate the object index map at the current moment, and form a depth map sequence and an object index map sequence.

[0010] Optionally, step five specifically includes: Read the depth map sequence and the object index map sequence sequentially according to time order, and traverse the corresponding pixel region for each object in the object index map at each time step to determine the object region of each object at the current time step. For each object, read the depth value distribution in the corresponding depth map of the object region at the current time, and count the visible pixel positions, region boundary positions, and region area in the object region to obtain the visible state result; Match the visible state results of adjacent time points according to the object identifier, perform region overlap comparison and depth continuity comparison on the object regions of the same object at adjacent time points, and determine adjacent time points where the overlapping area of ​​the object regions is greater than a preset ratio and the depth change is less than a preset threshold as the same continuous visible interval. Register the start and end times of the continuous visible intervals of the same object throughout all times to obtain the visible state intervals of each object, and determine the time period between two consecutive visible intervals where there is no object area or the object area is entirely in the occlusion position as the occlusion interval. Within the continuous visible intervals corresponding to each object, the region overlap comparison, boundary change comparison, and depth change comparison are performed on the object regions at adjacent times. The continuous times with region overlap area greater than a preset ratio, boundary change less than a preset threshold, and depth change less than a preset threshold are merged into a stable appearance segment. The object identifier, visible state range, occlusion range, and stable appearance segment of each object are associated and registered to establish an object continuity catalog.

[0011] Optionally, step six specifically includes: The visual Patch Tokens are arranged in order to form a visual Token sequence. Based on the noise intensity corresponding to the current diffusion time step, a random noise vector is superimposed on each visual Token in the visual Token sequence. The geometrically guided DiT model includes a visual input layer, a geometrically guided backbone layer, and a latent space output layer. The visual input layer includes a position encoding layer and a geometric condition encoder. The position encoding layer adds spatial position information to each visual token. The geometric condition encoder receives a depth map sequence and an object index map sequence, and encodes the depth distribution result and object region distribution result corresponding to the current time to generate a geometric feature sequence corresponding to the visual token sequence. The geometric guidance backbone layer consists of multiple Transformer Blocks arranged in series. Each Transformer Block includes an adaptive layer normalization unit, a visual self-attention unit, and a point-by-point feedforward network unit. The adaptive layer normalization unit receives the time step information corresponding to the diffusion time step and performs normalization modulation on the input features of the current layer. The visual self-attention unit interactively updates the texture features, color features, and contextual relationships between visual tokens. The point-by-point feedforward network unit performs dimensionality upscaling and projection on the visual features. A geometric cross-attention unit is arranged in series after the visual self-attention unit. In the geometric cross-attention unit, the query vector comes from the visual features updated by the visual self-attention unit, and the key vector and value vector come from the geometric feature sequence output by the geometric condition encoder. When the target object is detected to switch from an occluded state to a re-display state, the stable appearance fragment recorded in the object continuity directory is written into the geometric feature region corresponding to the current target object. The latent space output layer includes an Unpatchify layer and a linear output layer. The Unpatchify layer reassembles the output of the geometry-guided backbone layer according to the image block arrangement order to restore the latent space image features. The linear output layer performs a linear mapping on the latent space image features to obtain the video latent space feature sequence corresponding to the target video.

[0012] Optionally, step seven specifically includes: The latent space features at each time step in the video latent space feature sequence are decoded by the VAE decoder to obtain the video frame image at the corresponding time step. Temporal consistency correction is performed on video frame images at adjacent time points. The boundary position, texture distribution and color distribution of the corresponding object region in the video frame images at adjacent time points are compared. Boundary smoothing is performed on the position where the boundary change exceeds the preset threshold. Texture continuity correction is performed on the position where the texture change exceeds the preset threshold. Color consistency correction is performed on the position where the color change exceeds the preset threshold. After completing the timing consistency correction, the video frame images at each time point are arranged in chronological order, and the target video stream is output.

[0013] A deep learning-based video generation system according to an embodiment of the present invention includes the following modules: The input parsing and relation building module is used to obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; The relationship persistence encoding module is used to perform local state registration, consistency correction and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence feature items, and encode the relationship persistence feature items into a relationship persistence token sequence; The dynamic topology generation module is used to input the noisy, persistent relation token sequence into the topology-aware DiT model to generate a dynamic topology graph sequence. The geometric projection and index generation module is used to perform projective geometric projection and visibility layering calculations on dynamic topology graph sequences to generate depth map sequences and object index map sequences. The object continuity documentation module is used to perform occlusion persistence cataloging based on depth map sequences and object index map sequences, continuously register the visible state range and stable appearance segments of target objects, and establish an object continuity catalog. The latent space feature generation module is used to input the noisy visual Patch Token and object continuity catalog into the geometrically guided DiT model to obtain the video latent space feature sequence; The video decoding output module is used to input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding and output the target video stream.

[0014] The beneficial effects of this invention are: This invention addresses the problems of unstable multi-object interaction relationships and insufficient structural constraints caused by implicit hybrid modeling of object relationships in existing video generation technologies. This is achieved by introducing a set of technical features—layered topological scene decomposition, relation primitive construction, and relation persistence encoding—in the input stage. Text instructions and objects, relations, and their spatial correspondences in the reference image are explicitly organized into a scene-layered relation graph and a relation persistence token sequence. This allows subsequent generation processes to continuously evolve around object nodes and relation edges, improving the accuracy of structural representation in complex scenes. Furthermore, by introducing a graph node embedding layer, topological position encoding, topological sparse attention, and motion parameter decoder into the topology-aware DiT model, this invention solves the problems of standard DiT's inability to directly process graph topology sequences, the lack of physical connection constraints in fully connected attention, and the susceptibility to drift in long-term relation updates. This ensures that relation edge connection results and relation region position results remain stable during layer-by-layer denoising, significantly enhancing the spatiotemporal continuity of dynamic topological graph sequences. Finally, by performing projective geometric projection, visibility layer calculation, and Z-B mapping in the intermediate stage… Occlusion culling addresses the challenges of transforming 3D dynamic structures into stable 2D geometric constraints and accurately determining occlusion relationships. This results in a sequence of depth maps and object index maps, providing clear depth distribution and object boundary information for subsequent appearance generation. By performing persistent occlusion cataloging during the object continuity archiving stage, visible state intervals and stable appearance fragments are continuously registered, resolving the issues of existing methods lacking historical appearance recall information, identity drift, and texture flickering during target object re-emergence. Furthermore, by introducing a geometric conditional encoder and geometric cross-attention into the geometrically guided DiT model, depth constraints, object region constraints, and stable appearance fragments are jointly injected into the visual feature update process. This addresses the problem of standard DiT relying solely on visual context and lacking geometric constraints and re-emergence recovery capabilities, thereby improving the appearance consistency, boundary accuracy, and temporal coherence of generated videos in occlusion switching scenarios. Therefore, this invention is of great significance for improving structural stability, relational continuity, and the realism of re-emergence areas in long-term video generation. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a deep learning-based video generation method proposed in this invention; Figure 2 This is a framework diagram of the topology-aware DiT model in a deep learning-based video generation method proposed in this invention; Figure 3This is a framework diagram of the geometrically guided DiT model in a deep learning-based video generation method proposed in this invention; Figure 4 This is a framework diagram of a deep learning-based video generation system proposed in this invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0017] refer to Figures 1-3 A deep learning-based video generation method includes the following steps: Step 1: Obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; Step 2: Perform local state registration, consistency correction, and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence features, and encode the relationship persistence features into a relationship persistence token sequence; Step 3: Input the noisy relation persistence token sequence into the topology-aware DiT model. The topology-aware DiT model introduces a graph node embedding layer, topology position encoding, topology sparse attention and motion parameter decoder to generate a dynamic topology graph sequence. Step 4: Perform projective geometric projection and visibility layering calculations on the dynamic topology graph sequence to generate a depth map sequence and an object index map sequence; Step 5: Perform persistent occlusion cataloging based on the depth map sequence and object index map sequence, continuously register the visible state range and stable appearance fragments of the target object, and establish an object continuity catalog; Step 6: Input the noisy visual Patch Token and object continuity catalog into the geometry-guided DiT model. The geometry-guided DiT model introduces a geometric conditional encoder and geometric cross attention to obtain the video latent space feature sequence. Step 7: Input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding, and output the target video stream.

[0018] In this embodiment, step one specifically includes: The text instructions are parsed using the LTP dependency parser to extract object terms representing scene objects and relation terms representing the relationships between objects. Based on the parsing results, the modifiers and quantifiers that modify the same object term are merged to generate entity primitives. Relation terms that have dependency connections with two object terms are identified as relation primitives. The reference image is divided into regions using the SLIC superpixel segmentation algorithm to obtain multiple superpixel regions. The boundary information of each superpixel region is then extracted using the Canny edge detection operator. Combined with the connected component labeling algorithm, image regions with closed boundaries and connected component areas greater than a preset pixel threshold are identified as object candidate regions. Common boundary regions, overlapping boundary regions, and spaced adjacent regions located between adjacent object candidate regions are identified as relationship candidate regions. The bounding rectangle calculation, area calculation, center coordinate calculation, and contour closure determination are performed on the candidate regions of the objects. Based on the enclosing relationship between the bounding rectangles, the contact relationship between the region and the bottom or side edge of the image, and the area ratio of the region, the candidate regions of the objects with closed contours and not enclosed by other regions are determined as object regions, the candidate regions that contact the bottom or side edge of the image or form a continuous spreading area are determined as background regions, and the candidate regions located between the object region and the background region or the relationship between two object regions are determined as relationship regions. Alignment matching is performed between object terms and object regions in entity primitives according to the order of text appearance and image spatial position. Association mapping is performed between relation terms and relation regions in relation primitives according to the source object position and target object position. A scene hierarchical relationship graph is generated based on the alignment matching result and the association mapping result. The reference image is divided into image blocks, and each image block is encoded using Patch Embedding to generate a visual Patch Token that corresponds one-to-one with each image block.

[0019] In this embodiment, step two specifically includes: According to the numbering order of the object nodes in the scene hierarchical relationship graph, each object node is traversed in turn. All relation edges directly connected to the current object node are selected, and the relation region position corresponding to each relation edge is read. The current object node, relation edge and relation region position are combined into a local relation fragment. For each local relation segment, read the object identifier, object category, and region location of the object node; for each local relation segment, read the relation category, source object identifier, and target object identifier of the relation edge; and write the object identifier, object category, region location, relation category, source object identifier, target object identifier, and relation region location into the local state record result. The results of each local state record are classified according to the object identifier. Local state record results with the same object identifier are divided into the same comparison group, and within each comparison group, they are compared item by item according to whether the relational regions are adjacent or overlapping. For local state record results where object identifiers are inconsistent within the same comparison group, the order of occurrence of corresponding object terms in the text instruction is read, and the object identifier appearing earlier in the order is replaced with the object identifier appearing later in the order. For local state record results where relation categories are inconsistent within the same comparison group, the correspondence between relation region positions and relation terms is read, and the relation category corresponding to the relation region position is replaced with the relation category that does not correspond to the relation region position. For local state record results where relation region positions overlap within the same comparison group, the boundary contact ranges corresponding to the overlapping relation regions are compared, and the local state record results with larger boundary contact ranges are retained. For each comparison group after consistency correction, cross-fragment relationship propagation is performed according to the connection relationship between object nodes. The relationship category, target object identifier, and relationship region location in the current local state record result are written into the local state record result adjacent to the current object node, and the corresponding content in the adjacent local state record result is overwritten with the written relationship category, target object identifier, and relationship region location. After completing the cross-segment relationship propagation, the local state record results are sequentially concatenated according to the order of object identifier, relationship category, region location, and target object identifier to generate relationship persistence feature items; The persistent features of each relationship are sorted according to the connection order of the object nodes in the scene hierarchical relationship graph, and the sorted persistent features are input into the sequence encoding layer to be encoded into a persistent token sequence.

[0020] In this embodiment, step three specifically includes: According to the perturbation intensity corresponding to the current diffusion time step, a random noise component is superimposed on the relationship persistence token sequence; The topology-aware DiT model consists of an input processing layer, a core backbone layer, and an output processing layer: The input processing layer includes a graph node embedding layer, a topology location encoding unit, and a temporal step embedding unit: The graph node embedding layer receives the noisy relation persistence token sequence and converts each relation persistence token into a graph node token of a uniform dimension through linear mapping. The topology location encoding unit constructs a graph Laplacian matrix based on the scene hierarchical relation graph and performs feature decomposition on the graph Laplacian matrix. The obtained feature vector is selected as the relative position representation of the node in the topology structure, and then the relative position representation is superimposed on the corresponding graph node token. The temporal step embedding unit inputs the current diffusion time step into the multilayer perceptron and outputs a temporal step conditional vector. The core backbone layer consists of multiple cascaded Transformer Blocks. Each Transformer Block includes an adaptive layer normalization unit, a topological sparse attention unit, and a pointwise feedforward network unit. The adaptive layer normalization unit modulates the normalization result of the current layer's input features based on the time-step conditional vector. By dynamically adjusting the scaling and displacement components in the normalization result, it keeps the node feature distribution stable across different diffusion time steps. The topological sparse attention unit retains the projection methods of the query vector, key vector, and value vector, and introduces a topological binary mask matrix before attention normalization calculation. The topological binary mask matrix is ​​constructed based on the node connection relationships in the scene layer relationship graph and is used to indicate whether there is a direct connection between any two nodes. Only when a corresponding node pair has a connection relationship is it allowed to participate in the attention weight calculation, thus transforming the fully connected attention in standard DiT into sparse attention that is only performed between topologically connected nodes. The pointwise feedforward network unit receives the node features output by the topological sparse attention unit and sequentially passes them through the first linear layer, the GELU activation layer, and the second linear layer to perform nonlinear dimensionality upscaling and projection on the node features, obtaining the current Transformer. The feature update results of the Block are concatenated with multiple Transformer Blocks. The node hidden features output by the previous Transformer Block are used as the input of the next Transformer Block, so that the graph node features can be denoised and evolved layer by layer in the core backbone layer. The output processing layer includes a motion parameter decoder, which maps the node hidden features output from the last layer of the core backbone layer to the output results corresponding to the dynamic topology graph sequence. Specifically, the motion parameter decoder receives the node hidden features output from the last layer TransformerBlock and converts them into velocity parameters, displacement change parameters, and relation change parameters of object nodes and relation edges, respectively. Among them, the velocity parameters and displacement change parameters of object nodes represent the position change trend of object nodes between adjacent time steps, and the relation change parameters of relation edges represent the change trend of connection relationships between object nodes and the change trend of relation region positions. Based on the velocity parameters and displacement change parameters of object nodes, the spatiotemporal position results of object nodes at each time step are generated. Based on the relation change parameters of relation edges, the connection results of relation edges and the position results of relation regions at each time step are generated. Then, the spatiotemporal position results of object nodes, the connection results of relation edges, and the position results of relation regions are combined in chronological order to form a dynamic topology graph sequence.

[0021] In this embodiment, both the topology-aware DiT model and the original DiT model adopt the diffusion transformation framework of "input processing - backbone network denoising - output mapping", and retain the time step embedding, adaptive layer normalization, feedforward network and multi-layer Transformer Block cascaded calculation method. Unlike the original DiT model, this implementation introduces a graph node embedding layer and topological position encoding in the input processing layer for the graph topological data corresponding to the persistent token sequence of relationships. This replaces the original patch representation oriented towards image patches with a graph node representation oriented towards object nodes and relationship edges. In the core backbone layer, topological sparse attention is introduced, using a topological binary mask matrix to constrain attention interaction only between nodes with connections. In the output processing layer, a motion parameter decoder is introduced, transforming the output result from pixel noise prediction to object node motion parameters and relationship edge change parameters. The above improvements address the pain points of the original DiT model, such as its inability to directly process graph topology sequences, the lack of relational constraints in fully connected attention, and the tendency for long-term relational updates to drift. These improvements enable the connection relationships between object nodes, the location of relational regions, and the spatiotemporal evolution process to be continuously updated during layer-by-layer denoising, thereby improving the structural stability, relational continuity, and generation accuracy of dynamic topology graph sequences in complex interaction scenarios.

[0022] In this embodiment, step four specifically includes: The focal length parameters, principal point position parameters, and camera pose parameters are read according to the preset virtual camera parameters, and a geometric projection matrix is ​​constructed based on the focal length parameters, principal point position parameters, and camera pose parameters. Read the three-dimensional position results of the object nodes at each time point in the dynamic topology graph sequence in chronological order, and read the horizontal coordinate, vertical coordinate and depth coordinate of the three-dimensional position result of each object node. Input the horizontal, vertical, and depth coordinates of each object node into the geometric projection matrix, perform coordinate transformation and projection conversion, and obtain the projected coordinates of each object node in the two-dimensional image plane. Centered on the projection coordinates of each object node in the two-dimensional image plane, and based on the spatial distribution range of the object nodes at the current moment, the object projection area of ​​each object in the two-dimensional image plane is determined. The projection regions of each object in the two-dimensional image plane are traversed pixel by pixel, and the corresponding depth coordinates of multiple object projection regions located at the same pixel position are read. Compare the depth coordinates of the projected regions of each object at the same pixel location, determine the projected region of the object with the smaller depth coordinate as the front region, and determine the projected region of the object with the larger depth coordinate as the back region, thus completing the visibility layering calculation at that pixel location. Repeatedly perform depth coordinate comparison on all pixel positions in the two-dimensional image plane to obtain the front and back region distributions of the projection areas of each object in the entire image at the current moment. The Z-Buffer algorithm is applied to the front and back region distributions obtained by visibility layering calculation. At each pixel position, the depth value corresponding to the front region is retained, and the occluded part of the back region at that pixel position is deleted. Write the depth values ​​at each pixel position at the current time into the corresponding two-dimensional image position to generate the depth map at the current time. Write the object identifiers preserved at each pixel position at the current moment into the corresponding two-dimensional image positions to generate the object index map at the current moment; Repeatedly perform the following steps at all times in the dynamic topology map sequence: reading the 3D position of the object node, converting the geometric projection matrix, determining the object projection area, calculating the visibility layer, processing with the Z-Buffer algorithm, writing the depth value, and writing the object identifier. Arrange these steps in chronological order to form a depth map sequence and an object index map sequence.

[0023] In this embodiment, step five specifically includes: Read the depth map sequence and the object index map sequence sequentially according to time order, and traverse the corresponding pixel region for each object in the object index map at each time step to determine the object region of each object at the current time step. For each object, read the depth value distribution in the corresponding depth map of the object region at the current time, and count the visible pixel positions, region boundary positions, and region area in the object region to obtain the visible state result of the object at the current time. Match the visible state results of adjacent time points according to the object identifier, perform region overlap comparison and depth continuity comparison on the object regions of the same object at adjacent time points, and determine adjacent time points where the overlapping area of ​​the object regions is greater than a preset ratio and the depth change is less than a preset threshold as the same continuous visible interval. Register the start and end times of the continuous visible intervals of the same object throughout all times to obtain the visible state intervals of each object, and determine the time period between two consecutive visible intervals where there is no object area or the object area is entirely in the occlusion position as the occlusion interval. Within the continuous visible intervals corresponding to each object, the region overlap comparison, boundary change comparison, and depth change comparison are performed on the object regions at adjacent times. The continuous times with region overlap area greater than a preset ratio, boundary change less than a preset threshold, and depth change less than a preset threshold are merged into a stable appearance segment. The object identifier, visible state interval, occlusion interval, and stable appearance segment of each object are associated and registered to establish an object continuity directory. The object continuity directory shall at least record the object identifier, the start and end times of the visible state interval, the start and end times of the occlusion interval, and the time interval corresponding to the stable appearance segment.

[0024] In this embodiment, step six specifically includes: The visual Patch Tokens are arranged in order to form a visual Token sequence. Then, based on the noise intensity corresponding to the current diffusion time step, a random noise vector is superimposed on each visual Token in the visual Token sequence. The geometrically guided DiT model consists of a visual input layer, a geometrically guided backbone layer, and a latent space output layer. The visual input layer includes a position encoding layer and a geometric condition encoder. The position encoding layer uses a learnable position vector to attach its spatial position information in the image plane to each visual token. The geometric condition encoder receives a depth map sequence and an object index map sequence, and encodes the depth distribution result and object region distribution result corresponding to the current time to generate a geometric feature sequence corresponding to the visual token sequence. The geometrically guided backbone layer consists of multiple Transformer Blocks arranged in series. Each Transformer Block includes an adaptive layer normalization unit, a visual self-attention unit, and a point-by-point feedforward network unit. The adaptive layer normalization unit receives the time step information corresponding to the diffusion time step and performs normalization modulation on the input features of the current layer. The visual self-attention unit interactively updates the texture features, color features, and contextual relationships between visual tokens. The point-by-point feedforward network unit performs dimensionality upscaling and projection on the visual features through linear layers and nonlinear activation functions, thereby completing the feature update of the current layer. A geometric cross-attention unit is arranged in series after the visual self-attention unit. In the geometric cross-attention unit, the query vector comes from the visual features updated by the visual self-attention unit, and the key vector and value vector come from the geometric feature sequence output by the geometric conditional encoder, so that the visual features are simultaneously associated with the depth distribution and the boundary of the object region at the current time during the update process. In one embodiment, when the target object is detected to switch from an occluded state to a re-display state, the stable appearance fragment recorded in the object continuity catalog is written into the geometric feature region corresponding to the current target object, and participates in the update of the visual features corresponding to the current target object through the geometric cross-attention unit to complete the re-display anchoring recovery. The latent space output layer includes an Unpatchify layer and a linear output layer. The Unpatchify layer reassembles the visual token sequence output by the geometry-guided backbone layer according to the image patch arrangement order, restoring it to latent space image features. The linear output layer performs a linear mapping on the latent space image features to obtain the video latent space feature sequence corresponding to the target video.

[0025] In this embodiment, both the geometrically guided DiT model and the original DiT model adopt the processing framework of "visual input - backbone network denoising - latent space output", and retain position encoding, adaptive layer normalization, visual self-attention, point-by-point feedforward network, Unpatchify layer and linear output layer. Unlike the original DiT model, this implementation adds a geometric conditional encoder in addition to the standard visual input branch to encode the depth map sequence and the object index map sequence. Geometric cross-attention units are set in series in the backbone network to inject constraints using visual features as queries and geometric features as keys and values. At the same time, when the target object switches from an occluded state to a re-revealed state, stable appearance fragments in the object continuity catalog are written into the corresponding geometric feature regions to participate in the visual feature update of the current object region.

[0026] The above improvements address the shortcomings of the original DiT model, which relies solely on visual context and lacks explicit geometric constraints and re-recovery criteria. This allows the generation process to be constrained by depth distribution, object region boundaries, and historically stable appearance fragments simultaneously, thereby reducing boundary misalignment, texture drift, and identity inconsistency issues in occlusion switching scenarios and improving the geometric consistency, appearance continuity, and re-recovery region realism of video latent space feature sequences.

[0027] In this embodiment, step seven specifically includes: The latent space features at each time step in the video latent space feature sequence are decoded by the VAE decoder to obtain the video frame image at the corresponding time step. Temporal consistency correction is performed on video frame images at adjacent time points. The boundary position, texture distribution and color distribution of the corresponding object region in the video frame images at adjacent time points are compared. Boundary smoothing is performed on the position where the boundary change exceeds the preset threshold. Texture continuity correction is performed on the position where the texture change exceeds the preset threshold. Color consistency correction is performed on the position where the color change exceeds the preset threshold. After completing the timing consistency correction, the video frame images at each time point are arranged in chronological order, and the target video stream is output.

[0028] refer to Figure 4 A deep learning-based video generation system includes the following modules: The input parsing and relation building module is used to obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; The relationship persistence encoding module is used to perform local state registration, consistency correction and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence feature items, and encode the relationship persistence feature items into a relationship persistence token sequence; The dynamic topology generation module is used to input the noisy, persistent relation token sequence into the topology-aware DiT model to generate a dynamic topology graph sequence. The geometric projection and index generation module is used to perform projective geometric projection and visibility layering calculations on dynamic topology graph sequences to generate depth map sequences and object index map sequences. The object continuity documentation module is used to perform occlusion persistence cataloging based on depth map sequences and object index map sequences, continuously register the visible state range and stable appearance segments of target objects, and establish an object continuity catalog. The latent space feature generation module is used to input the noisy visual Patch Token and object continuity catalog into the geometrically guided DiT model to obtain the video latent space feature sequence; The video decoding output module is used to input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding and output the target video stream.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to a scenario of automatically generating short advertising videos for nighttime commercial streets in an urban area. In this scenario, the operator inputs the text command "Generate a 6-second video: A female customer enters the commercial street from the left, stops in front of a coffee shop, then turns to look at the shop window on the right. A delivery truck slowly drives by in the frame, and the streetlights and shadows change slightly with the person's movement," while simultaneously inputting a reference image of the commercial street as the appearance benchmark. The challenge of this scenario lies in the continuous occlusion and re-emphasis relationships between people, vehicles, shop signs, and shop windows. If traditional text-to-video methods are used, problems such as flickering edges of the person's arms, sudden changes in reflections in the shop window when a vehicle passes by, and drifting of clothing folds and facial details after the person turns around can easily occur.

[0030] On the text side, the LTP dependency parser was used to extract entity primitives and relation primitives. On the image side, the SLIC superpixel segmentation algorithm was used, with 280 superpixels and a compactness of 18. Edge detection employed the Canny operator, with high and low thresholds set to 120 and 60, respectively. The connected component area threshold was set to 350 pixels. The reference image resolution was set to 768×512, the image patch size to 16×16, and 1536 visual patch tokens were generated. The relation persistence token sequence length was set to 96, and the diffusion steps to 50. The topology-aware DiT backbone layer consisted of 12 Transformer Blocks, 8 attention heads, and a hidden dimension of 768; the geometry-guided DiT backbone layer consisted of 16 Transformer Blocks, 12 attention heads, and a hidden dimension of 1024. The video output was 24 frames per second, for a total of 144 frames, with a latent space size of 96×64. In the object continuity catalog, the threshold for determining continuous visible intervals is set to an overlap ratio of no less than 0.72 between adjacent frame regions, the threshold for depth change is set to 0.08, and the threshold for merging stable appearance fragments is set to a boundary change of no more than 5 pixels.

[0031] In practical applications, the main entities and relation primitives are first parsed from text instructions, and then combined with reference images to form a scene-layered relation graph. Subsequently, consistency correction and cross-fragment propagation are performed on local relation fragments to generate a persistent relation token sequence. This persistent relation token sequence is then input into a topology-aware DiT model. Under the influence of graph node embedding layers and topological position encoding, the relationships between people-vehicles, people-shop windows, and vehicles-shop windows are denoised and dynamically evolved layer by layer, resulting in a dynamic topology graph sequence. Next, based on preset virtual camera parameters (focal length 35mm, principal point position set to the image center, camera tilt angle set to 4 degrees), projective geometric projection and visibility layering calculations are performed on the dynamic topology graph to generate a depth map sequence and an object index map sequence. Furthermore, the depth map sequence and object index map sequence are used to establish an object continuity catalog, cataloging stable appearance fragments such as people's faces, clothing textures, vehicle side markings, and shop window highlight areas. Finally, the noisy visual patch token and object continuity catalog are input into the geometry-guided DiT model. Under the constraints of geometric conditional encoder and geometric cross-attention, video latent space features are generated, and then the target video stream is output through VAE decoder. To demonstrate the beneficial effects of this invention, this embodiment also sets up three sets of comparison methods. The first set is the traditional text-to-video diffusion method A, which does not use scene layer relationship graph, object continuity catalog, and geometric conditional injection; the second set is the basic DiT video generation method B, which uses reference images but does not introduce the improved mechanisms of topology-aware DiT and geometry-guided DiT; the third set is the simplified version C of this invention, which only replaces the improved model but does not perform occlusion persistence cataloging and object continuity catalog construction; the fourth set is the method of this invention.

[0032] To ensure comparability, all four methods were run 10 times on the same dataset, with the same prompts and the same output length. Structural consistency, occlusion-re-display consistency, temporal stability, user satisfaction, and average inference time were statistically analyzed. All scores were out of 100. Structural consistency was scored by three video algorithm engineers on the stability of the person-vehicle-storefront relationship; occlusion-re-display consistency was scored based on the accuracy of appearance restoration after occlusion; temporal stability was scored based on the degree of flickering and boundary jumps between adjacent frames; and user satisfaction was scored by five content creators. The results are shown in Table 1 below. Table 1. Overall Performance Comparison of Different Video Generation Methods

[0033] Further statistical analysis was conducted on the quantitative results of key objects. Four key observation areas were identified: facial areas, clothing texture areas, vehicle side marking areas, and shop window highlight areas. Regional consistency error (index 1), first re-display error after occlusion (index 2), and cross-frame brightness fluctuation value (index 3) were statistically analyzed. The results are shown in Table 2 below: Table 2. Statistics on the Consistency and Occlusion Re-display Effect of Key Object Areas

[0034] As shown in Tables 1 and 2, traditional method A performs the worst in terms of structural consistency and occlusion-re-reveal consistency, with 19 edge transitions for characters and 14 texture drifts for vehicles, indicating that it relies solely on unified latent space generation and cannot effectively handle multi-object interactions and occlusion transitions. Basic DiT method B shows improvement over A, indicating that Transformer-based visual modeling has a positive effect on texture realism and contextual relationships. However, without topological constraints and an object continuity catalog, characters still exhibit clothing drift and abrupt changes in window reflections after being occluded by vehicles. Simplified version C significantly outperforms B in terms of structural consistency and temporal stability, demonstrating that topology-aware DiT and geometry-guided DiT can enhance scene relationship modeling and geometric constraint capabilities. However, due to the lack of an object continuity catalog, the initial re-reveal error after occlusion remains high. The complete method D achieved the best results across all metrics, particularly in occlusion re-display consistency, reaching 91.8 points, a 19.4-point improvement over the basic DiT method B. The number of edge jumps in people decreased from 13 to 3, and the number of texture drifts in vehicle re-display decreased from 9 to 1. The average regional consistency error in the four key observation areas decreased by approximately 59%, and the average brightness fluctuation value decreased by approximately 64%. This demonstrates that the invention not only improves image clarity but also achieves synergistic improvements in structural stability, accurate occlusion recovery, and cross-frame appearance continuity.

[0035] As can be seen from this embodiment, this invention, through scene-layered relationship graphs and persistent relationship token sequences, enables objects and relationships in complex scenes to be structurally represented before entering the generation process, solving the structural instability problem caused by implicit hybrid modeling of object relationships in traditional methods. Through graph node embedding layers, topological position encoding, and topological sparse attention in the topology-aware DiT model, the edges between people, vehicles, and shop windows can evolve according to real-world connections, reducing relationship jumps in long-term generation. Through projective geometric projection, visibility layering, and persistent occlusion cataloging, target objects can recover appearance details by calling stable appearance fragments after occlusion. Furthermore, through the geometrically guided DiT model's geometric conditional encoder and geometric cross-attention, depth constraints, object region boundaries, and historical appearance anchoring are simultaneously injected into the visual generation process, thereby achieving higher structural consistency, temporal stability, and re-enhancing realism. Therefore, this invention can be effectively applied to business scenarios such as commercial advertising short videos, digital human demonstration videos, and automatically generated videos for retail scenes, demonstrating good engineering feasibility and significant technical advantages.

[0036] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A video generation method based on deep learning, characterized in that, Includes the following steps: Step 1: Obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; Step 2: Perform local state registration, consistency correction, and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence features, and encode the relationship persistence features into a relationship persistence token sequence; Step 3: Input the noisy relation persistence token sequence into the topology-aware DiT model. The topology-aware DiT model introduces a graph node embedding layer, topology position encoding, topology sparse attention and motion parameter decoder to generate a dynamic topology graph sequence. Step 4: Perform projective geometric projection and visibility layering calculations on the dynamic topology graph sequence to generate a depth map sequence and an object index map sequence; Step 5: Perform persistent occlusion cataloging based on the depth map sequence and object index map sequence, continuously register the visible state range and stable appearance fragments of the target object, and establish an object continuity catalog; Step 6: Input the noisy visual Patch Token and object continuity catalog into the geometry-guided DiT model, which introduces a geometric conditional encoder and geometric cross attention to obtain the video latent space feature sequence; Step 7: Input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding, and output the target video stream.

2. The video generation method based on deep learning according to claim 1, characterized in that, Step one specifically includes: The text instructions are parsed syntactically to extract object terms representing scene objects and relation terms representing the relationships between objects. Based on the syntactic parsing results, the attributive and quantifier words modifying the same object term are merged to generate entity primitives. Relation terms that have dependency connections with two object terms are identified as relation primitives. The reference image is divided into regions to obtain multiple superpixel regions. The Canny edge detection operator is used to extract the boundary information of each superpixel region. Image regions with closed boundaries and connected component areas greater than a preset pixel threshold are identified as object candidate regions. Common boundary regions, overlapping boundary regions, and adjacent regions located between adjacent object candidate regions are identified as relation candidate regions. The object candidate regions are divided into object regions, background regions, and relation regions. Alignment matching is performed between object terms in entity primitives and object regions, and association mapping is performed between relation terms in relation primitives and relation regions. Object nodes are generated from matched object regions, and relation edges are generated from mapped relation regions, thus generating a scene hierarchical relation graph. The reference image is divided and encoded into image blocks to generate a visual Patch Token that corresponds one-to-one with each image block.

3. The video generation method based on deep learning according to claim 1, characterized in that, Step two specifically includes: Traverse each object node in the scene hierarchical relationship graph, select all relationship edges that are directly connected to the current object node, read the relationship region corresponding to each relationship edge, and combine the current object node, relationship edge, and relationship region into a local relationship fragment. For each local relation segment, read the object identifier, object category, and region location of the object node; for each local relation segment, read the relation category, source object identifier, and target object identifier of the relation edge; and write the results into the local state record. The results of each local state record are classified according to the object identifier. Local state record results with the same object identifier are divided into the same comparison group, and consistency correction is performed within each comparison group. For each comparison group after consistency correction, cross-fragment relationship propagation is performed according to the connection relationship between object nodes. The relationship category, target object identifier, and relationship region location in the current local state record result are written into the local state record result adjacent to the current object node, and the corresponding content in the adjacent local state record result is overwritten with the written relationship category, target object identifier, and relationship region location. After completing the cross-segment relationship propagation, the local state record results are sequentially concatenated according to the order of object identifier, relationship category, region location, and target object identifier to generate relationship persistence feature items; The persistent features of each relationship are sorted according to the connection order of the object nodes in the scene hierarchical relationship graph, and the sorted persistent features are input into the sequence encoding layer and encoded into a persistent token sequence.

4. The video generation method based on deep learning according to claim 1, characterized in that, Step three specifically includes: According to the perturbation intensity corresponding to the current diffusion time step, a random noise component is superimposed on the relationship persistence token sequence; The topology-aware DiT model includes an input processing layer, a core backbone layer, and an output processing layer: The input processing layer includes a graph node embedding layer, a topology location encoding unit, and a time step embedding unit: the graph node embedding layer maps the relationship persistence token sequence to graph node tokens; the topology location encoding unit constructs a graph Laplacian matrix based on the scene hierarchical relationship graph, performs feature decomposition on the graph Laplacian matrix, selects the obtained feature vector as the relative position representation of the node in the topology, and superimposes the relative position representation into the corresponding graph node token; the time step embedding unit inputs the current diffusion time step into the multilayer perceptron and outputs a time step condition vector. The core backbone layer consists of multiple Transformer Blocks arranged in series. Each Transformer Block includes an adaptive layer normalization unit, a topological sparse attention unit, and a point-by-point feedforward network unit. The adaptive layer normalization unit modulates the normalization result of the graph node tokens input to the current layer according to the time step condition vector. The topological sparse attention unit introduces a topological binary mask matrix, which is constructed based on the node connection relationship in the scene layer relationship graph. The topological binary mask matrix is ​​only allowed to participate in the attention weight calculation when the corresponding node pairs have a connection relationship. The point-by-point feedforward network unit performs nonlinear dimensionality upscaling and projection on the node features output by the topological sparse attention unit to obtain the feature update result of the current Transformer Block. The output processing layer includes a motion parameter decoder. The motion parameter decoder receives the node hidden features output by the last TransformerBlock layer and converts the node hidden features into velocity parameters, displacement change parameters, and relation change parameters of the object nodes and their edges, respectively. Based on the velocity parameters and displacement change parameters of the object nodes, the spatiotemporal position results of the object nodes at each time moment are generated. Based on the relation change parameters of the relation edges, the connection results of the relation edges and the position results of the relation regions at each time moment are generated. The spatiotemporal position results of the object nodes, the connection results of the relation edges, and the position results of the relation regions are combined in chronological order to form a dynamic topology graph sequence.

5. The video generation method based on deep learning according to claim 1, characterized in that, Step four specifically includes: The focal length parameters, principal point position parameters, and camera pose parameters are read according to the preset virtual camera parameters, and a geometric projection matrix is ​​constructed based on the focal length parameters, principal point position parameters, and camera pose parameters. Read the three-dimensional position results of the object nodes at each time point in the dynamic topology graph sequence in chronological order, and read the horizontal coordinate, vertical coordinate and depth coordinate of the three-dimensional position result of each object node. Input the horizontal, vertical, and depth coordinates of each object node into the geometric projection matrix, perform coordinate transformation and projection conversion, and obtain the projected coordinates of each object node in the two-dimensional image plane. Centered on the projection coordinates of each object node in the two-dimensional image plane, and based on the spatial distribution range of the object nodes at the current moment, the object projection area of ​​each object in the two-dimensional image plane is determined. The projection regions of each object in the two-dimensional image plane are traversed pixel by pixel, and the corresponding depth coordinates of multiple object projection regions located at the same pixel position are read. Compare the depth coordinates of the projected areas of each object at the same pixel position, and determine the projected area of ​​the object with the smaller depth coordinate as the front area and the projected area of ​​the object with the larger depth coordinate as the back area. Repeatedly perform depth coordinate comparison on all pixel positions in the two-dimensional image plane to obtain the front and back region distributions of the projection areas of each object in the entire image at the current moment. The Z-Buffer algorithm is applied to the front and back region distributions obtained by visibility layering calculation. At each pixel position, the depth value corresponding to the front region is retained, and the occluded part of the back region at the pixel position is deleted. Write the depth values ​​at each pixel position at the current time into the corresponding two-dimensional image position to generate the depth map at the current time. Write the object identifiers preserved at each pixel position at the current moment into the corresponding two-dimensional image position to generate the object index map at the current moment, and form a depth map sequence and an object index map sequence.

6. The video generation method based on deep learning according to claim 1, characterized in that, Step five specifically includes: Read the depth map sequence and the object index map sequence sequentially according to time order, and traverse the corresponding pixel region for each object in the object index map at each time step to determine the object region of each object at the current time step. For each object, read the depth value distribution in the corresponding depth map of the object region at the current time, and count the visible pixel positions, region boundary positions, and region area in the object region to obtain the visible state result; Match the visible state results of adjacent time points according to the object identifier, perform region overlap comparison and depth continuity comparison on the object regions of the same object at adjacent time points, and determine adjacent time points where the overlapping area of ​​the object regions is greater than a preset ratio and the depth change is less than a preset threshold as the same continuous visible interval. Register the start and end times of the continuous visible intervals of the same object throughout all times to obtain the visible state intervals of each object, and determine the time period between two consecutive visible intervals where there is no object area or the object area is entirely in the occlusion position as the occlusion interval. Within the continuous visible intervals corresponding to each object, the region overlap comparison, boundary change comparison, and depth change comparison are performed on the object regions at adjacent times. The continuous times with region overlap area greater than a preset ratio, boundary change less than a preset threshold, and depth change less than a preset threshold are merged into a stable appearance segment. The object identifier, visible state range, occlusion range, and stable appearance segment of each object are associated and registered to establish an object continuity catalog.

7. The video generation method based on deep learning according to claim 1, characterized in that, Step six specifically includes: The visual Patch Tokens are arranged in order to form a visual Token sequence. Based on the noise intensity corresponding to the current diffusion time step, a random noise vector is superimposed on each visual Token in the visual Token sequence. The geometrically guided DiT model includes a visual input layer, a geometrically guided backbone layer, and a latent space output layer. The visual input layer includes a position encoding layer and a geometric condition encoder. The position encoding layer adds spatial position information to each visual token. The geometric condition encoder receives a depth map sequence and an object index map sequence, and encodes the depth distribution result and object region distribution result corresponding to the current time to generate a geometric feature sequence corresponding to the visual token sequence. The geometric guidance backbone layer consists of multiple Transformer Blocks arranged in series. Each Transformer Block includes an adaptive layer normalization unit, a visual self-attention unit, and a point-by-point feedforward network unit. The adaptive layer normalization unit receives the time step information corresponding to the diffusion time step and performs normalization modulation on the input features of the current layer. The visual self-attention unit interactively updates the texture features, color features, and contextual relationships between visual tokens. The point-by-point feedforward network unit performs dimensionality upscaling and projection on the visual features. A geometric cross-attention unit is arranged in series after the visual self-attention unit. In the geometric cross-attention unit, the query vector comes from the visual features updated by the visual self-attention unit, and the key vector and value vector come from the geometric feature sequence output by the geometric condition encoder. When the target object is detected to switch from an occluded state to a re-display state, the stable appearance fragment recorded in the object continuity directory is written into the geometric feature region corresponding to the current target object. The latent space output layer includes an Unpatchify layer and a linear output layer. The Unpatchify layer reassembles the output of the geometry-guided backbone layer according to the image block arrangement order to restore the latent space image features. The linear output layer performs a linear mapping on the latent space image features to obtain the video latent space feature sequence corresponding to the target video.

8. The video generation method based on deep learning according to claim 1, characterized in that, Step seven specifically includes: The latent space features at each time step in the video latent space feature sequence are decoded by the VAE decoder to obtain the video frame image at the corresponding time step. Temporal consistency correction is performed on video frame images at adjacent time points. The boundary position, texture distribution and color distribution of the corresponding object region in the video frame images at adjacent time points are compared. Boundary smoothing is performed on the position where the boundary change exceeds the preset threshold. Texture continuity correction is performed on the position where the texture change exceeds the preset threshold. Color consistency correction is performed on the position where the color change exceeds the preset threshold. After completing the timing consistency correction, the video frame images at each time point are arranged in chronological order, and the target video stream is output.

9. A deep learning-based video generation system, comprising executing the deep learning-based video generation method according to any one of claims 1 to 8, characterized in that, Includes the following modules: The input parsing and relation building module is used to obtain input content containing text instructions and reference images, perform layered topological scene decomposition and relation primitive construction on the input content, extract entity primitives and relation primitives, generate scene layered relation graph, and generate visual Patch Token based on reference images; The relationship persistence encoding module is used to perform local state registration, consistency correction and cross-fragment relationship propagation on the scene hierarchical relationship graph and local relationship fragments, construct relationship persistence feature items, and encode the relationship persistence feature items into a relationship persistence token sequence; The dynamic topology generation module is used to input the noisy, persistent relation token sequence into the topology-aware DiT model to generate a dynamic topology graph sequence. The geometric projection and index generation module is used to perform projective geometric projection and visibility layering calculations on dynamic topology graph sequences to generate depth map sequences and object index map sequences. The object continuity documentation module is used to perform occlusion persistence cataloging based on depth map sequences and object index map sequences, continuously register the visible state range and stable appearance segments of target objects, and establish an object continuity catalog. The latent space feature generation module is used to input the noisy visual Patch Token and object continuity catalog into the geometrically guided DiT model to obtain the video latent space feature sequence; The video decoding output module is used to input the video latent space feature sequence into the VAE decoder for spatiotemporal consistency decoding and output the target video stream.

Images